Merge pull request #5185 from Security-Onion-Solutions/dev

2.3.70

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Contributing to Security Onion
+
+### Questions, suggestions, and general comments
+* Security Onion uses GitHub's [Discussions](https://github.com/Security-Onion-Solutions/securityonion/discussions) to provide a forum where the community and developers can interact as well as ask and answer questions.
+
+### Reporting a bug
+* The primary place to report unexpected behavior or possible bugs is the repo's [Discussions forum](https://github.com/Security-Onion-Solutions/securityonion/discussions).
+
+*  **If you are familiar with the current version of Security Onion and are confident you've discovered a bug**, first ensure there is not already an issue present by searching the open [issues](https://github.com/Security-Onion-Solutions/securityonion/issues). If there is, a thumbs up :+1: is a great way to show this bug is affecting you too.
+
+* If an issue doesn't exist, [open a new one](https://github.com/Security-Onion-Solutions/securityonion/issues/new), following the directions in the issue template. This means including:
+  * **System information** and how Security Onion was installed
+  * **Log files** relevant to the bug report
+  * **Reproduction steps** 
+
+### Contributing code
+
+* **All commits must be signed** with a valid key that has been added to your GitHub account. The commits should have all the "**Verified**" tag when viewed on GitHub as shown below:
+  
+  <img src="./assets/images/verified-commit-1.png" width="450">
+
+* If an issue does not already exist for the bug or feature for which you are submitting a pull request, [create one](https://github.com/Security-Onion-Solutions/securityonion/issues/new) with the relevant prefix. (**`FIX:`** for bug fixes, **`FEATURE:`** for new features.)
+
+* Link the PR to the related issue, either using [keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) in the PR description, or [manually](https://docs.github.com/en/issues/tracking-your-work-with-issues/creating-issues/linking-a-pull-request-to-an-issue#manually-linking-a-pull-request-to-an-issue).
+
+* **Pull requests should be opened against the `dev` branch of this repo**, and should clearly describe the problem and solution.
+
+* Be sure you have tested your changes and are confident they will not break other parts of the product.
+
+* See this document's [code styling and conventions section](#code-style-and-conventions) below to be sure your PR fits our code requirements prior to submitting.
+
+
+
+### Code style and conventions
+* **Keep code [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)**. For example, Bash code used by multiple scripts will likely best be added to <span style="white-space: nowrap;">[`so-common`](salt/common/tools/sbin/so-common)</span>.
+
+* All new Bash code should pass [ShellCheck](https://www.shellcheck.net/) analysis. Where errors can be *safely* [ignored](https://github.com/koalaman/shellcheck/wiki/Ignore), the relevant disable directive should be accompanied by a brief explanation as to why the error is being ignored.
+
+* **Ensure all YAML (this includes Salt states and pillars) is properly formatted**. The spec for YAML v1.2 can be found [here](https://yaml.org/spec/1.2/spec.html), however there are numerous online resources with simpler descriptions of its formatting rules. 

HOTFIX

@@ -0,0 +1 @@
+

README.md

@@ -1,14 +1,14 @@
-## Security Onion 2.3.30
+## Security Onion 2.3.70
 
-Security Onion 2.3.30 is here!
+Security Onion 2.3.70 is here!
 
 ## Screenshots
 
 Alerts
-![Alerts](https://raw.githubusercontent.com/security-onion-solutions/securityonion/master/screenshots/alerts-1.png)
+![Alerts](./assets/images/screenshots/alerts-1.png)
 
 Hunt
-![Hunt](https://raw.githubusercontent.com/security-onion-solutions/securityonion/master/screenshots/hunt-1.png)
+![Hunt](./assets/images/screenshots/hunt-1.png)
 
 ### Release Notes
 

SECURITY.md

@@ -0,0 +1,21 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x.x   | :white_check_mark: |
+| 16.04.x | :x:                |
+
+Security Onion 16.04 has reached End Of Life and is no longer supported.
+
+## Reporting a Vulnerability
+
+If you have any security concerns regarding Security Onion or believe you have uncovered a vulnerability, please follow these steps:
+
+- send an email to security@securityonion.net
+- include a description of the issue and steps to reproduce
+- please use plain text format (no Word documents or PDF files)
+- please do not disclose publicly until we have had sufficient time to resolve the issue
+
+This security address should be used only for undisclosed vulnerabilities. Dealing with fixed issues or general questions on how to use Security Onion should be handled via the normal support channels.

VERIFY_ISO.md

@@ -1,16 +1,18 @@
-### 2.3.30 ISO image built on 2021/03/01
+### 2.3.70 ISO image built on 2021/08/17
+
+
 
 ### Download and Verify
 
-2.3.30 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-2.3.30.iso
+2.3.70 ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-2.3.70.iso
 
-MD5: 65202BA0F7661A5E27087F097B8E571E  
-SHA1: 14E842E39EDBB55A104263281CF25BF88A2E9D67  
-SHA256: 210B37B9E3DFC827AFE2940E2C87B175ADA968EDD04298A5926F63D9269847B7 
+MD5: F048FABC7FD2D0E1A8B02381F115D1E0  
+SHA1: DF6D20FEF13CDC1B19309D2A1178D6E5D25FDA6F  
+SHA256: B193FFD7EE69958A8E257117149DCFB2125C5772FBFA6003AD80FD1CC129E571 
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.30.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.70.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS  
@@ -24,22 +26,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/ma
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.30.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.70.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-2.3.30.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-2.3.70.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-2.3.30.iso.sig securityonion-2.3.30.iso
+gpg --verify securityonion-2.3.70.iso.sig securityonion-2.3.70.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Mon 01 Mar 2021 02:15:28 PM EST using RSA key ID FE507013
+gpg: Signature made Tue 17 Aug 2021 10:52:17 PM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

VERSION

@@ -1 +1 @@
-2.3.30
+2.3.70

files/salt/master/master

@@ -67,3 +67,7 @@ peer:
 reactor:
   - 'so/fleet':
     - salt://reactor/fleet.sls
+  - 'salt/beacon/*/watch_sqlite_db//opt/so/conf/kratos/db/sqlite.db':
+    - salt://reactor/kratos.sls
+
+

pillar/docker/config.sls

@@ -1,208 +0,0 @@
-{%- set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
-{%- set FLEETNODE = salt['pillar.get']('global:fleet_node', False) -%}
-{% set WAZUH = salt['pillar.get']('manager:wazuh', '0') %}
-{% set THEHIVE = salt['pillar.get']('manager:thehive', '0') %}
-{% set PLAYBOOK = salt['pillar.get']('manager:playbook', '0') %}
-{% set FREQSERVER = salt['pillar.get']('manager:freq', '0') %}
-{% set DOMAINSTATS = salt['pillar.get']('manager:domainstats', '0') %}
-{% set ZEEKVER = salt['pillar.get']('global:mdengine', 'COMMUNITY') %}
-{% set GRAFANA = salt['pillar.get']('manager:grafana', '0') %}
-
-eval:
-  containers:
-    - so-nginx
-    - so-telegraf
-    {% if  GRAFANA == '1' %}
-    - so-influxdb
-    - so-grafana
-    {% endif %}
-    - so-dockerregistry
-    - so-soc
-    - so-kratos
-    - so-idstools
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    - so-elasticsearch
-    - so-logstash
-    - so-kibana
-    - so-steno
-    - so-suricata
-    - so-zeek
-    - so-curator
-    - so-elastalert
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-heavy_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-redis
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-steno
-    - so-suricata
-    - so-wazuh
-    - so-filebeat
-    {% if ZEEKVER != 'SURICATA' %}
-    - so-zeek
-    {% endif %}
-helix:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-idstools
-    - so-steno
-    - so-zeek
-    - so-redis
-    - so-logstash
-    - so-filebeat
-hot_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-manager_search:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-soc
-    - so-kratos
-    - so-acng
-    - so-idstools
-    - so-redis
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-kibana
-    - so-elastalert
-    - so-filebeat
-    - so-soctopus
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-manager:
-  containers:
-    - so-dockerregistry
-    - so-nginx
-    - so-telegraf
-    {% if  GRAFANA == '1' %}
-    - so-influxdb
-    - so-grafana
-    {% endif %}
-    - so-soc
-    - so-kratos
-    - so-acng
-    - so-idstools
-    - so-redis
-    - so-elasticsearch
-    - so-logstash
-    - so-kibana
-    - so-elastalert
-    - so-filebeat
-    {% if FLEETMANAGER %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    {% endif %}
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-    - so-soctopus
-    {% if THEHIVE != '0' %}
-    - so-thehive
-    - so-thehive-es
-    - so-cortex
-    {% endif %}
-    {% if PLAYBOOK != '0' %}
-    - so-playbook
-    {% endif %}
-    {% if FREQSERVER != '0' %}
-    - so-freqserver
-    {% endif %}
-    {% if DOMAINSTATS != '0' %}
-    - so-domainstats
-    {% endif %}
-parser_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-search_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-logstash
-    - so-elasticsearch
-    - so-curator
-    - so-filebeat
-    {% if WAZUH != '0' %}
-    - so-wazuh
-    {% endif %}
-sensor:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-steno
-    - so-suricata
-    {% if ZEEKVER != 'SURICATA' %}
-    - so-zeek
-    {% endif %}
-    - so-wazuh
-    - so-filebeat
-warm_node:
-  containers:
-    - so-nginx
-    - so-telegraf
-    - so-elasticsearch
-fleet:
-  containers:
-    {% if FLEETNODE %}
-    - so-mysql
-    - so-fleet
-    - so-redis
-    - so-filebeat
-    - so-nginx
-    - so-telegraf
-    {% endif %}

pillar/logstash/search.sls

@@ -7,8 +7,10 @@ logstash:
         - so/9000_output_zeek.conf.jinja
         - so/9002_output_import.conf.jinja
         - so/9034_output_syslog.conf.jinja
-        - so/9100_output_osquery.conf.jinja
+        - so/9050_output_filebeatmodules.conf.jinja
+        - so/9100_output_osquery.conf.jinja  
         - so/9400_output_suricata.conf.jinja
         - so/9500_output_beats.conf.jinja
         - so/9600_output_ossec.conf.jinja
         - so/9700_output_strelka.conf.jinja
+        - so/9800_output_logscan.conf.jinja

pillar/top.sls

@@ -22,6 +22,9 @@ base:
   '*_manager or *_managersearch':
     - match: compound
     - data.*
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - secrets
     - global
     - minions.{{ grains.id }}
@@ -38,6 +41,9 @@ base:
     - secrets
     - healthcheck.eval
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - global
     - minions.{{ grains.id }}
 
@@ -46,6 +52,9 @@ base:
     - logstash.manager
     - logstash.search
     - elasticsearch.search
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - data.*
     - zeeklogs
     - secrets
@@ -59,6 +68,7 @@ base:
 
   '*_heavynode':
     - zeeklogs
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
 
@@ -80,6 +90,7 @@ base:
     - logstash
     - logstash.search
     - elasticsearch.search
+    - elasticsearch.auth
     - global
     - minions.{{ grains.id }}
     - data.nodestab
@@ -88,5 +99,8 @@ base:
     - zeeklogs
     - secrets
     - elasticsearch.eval
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
+    - elasticsearch.auth
+{% endif %}
     - global
-    - minions.{{ grains.id }}
+    - minions.{{ grains.id }}

pillar/zeek/init.sls

@@ -52,5 +52,4 @@ zeek:
       - frameworks/signatures/detect-windows-shells
     redef:
       - LogAscii::use_json = T;
-      - LogAscii::json_timestamps = JSON::TS_ISO8601;
-      - CaptureLoss::watch_interval = 5 mins;
+      - CaptureLoss::watch_interval = 5 mins;

salt/airgap/init.sls

@@ -1,71 +0,0 @@
-{% from 'allowed_states.map.jinja' import allowed_states %}
-{% if sls in allowed_states %}
-
-{% set MANAGER = salt['grains.get']('master') %}
-airgapyum:
-  file.managed:
-    - name: /etc/yum/yum.conf
-    - source: salt://airgap/files/yum.conf
-
-airgap_repo:
-  pkgrepo.managed:
-    - humanname: Airgap Repo
-    - baseurl: https://{{ MANAGER }}/repo
-    - gpgcheck: 0
-    - sslverify: 0
-
-agbase:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Base.repo
-
-agcr:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-CR.repo
-
-agdebug:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Debuginfo.repo
-
-agfasttrack:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-fasttrack.repo
-
-agmedia:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Media.repo
-
-agsources:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Sources.repo
-
-agvault:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-Vault.repo
-
-agkernel:
-  file.absent:
-    - name: /etc/yum.repos.d/CentOS-x86_64-kernel.repo
-
-agepel:
-  file.absent:
-    - name: /etc/yum.repos.d/epel.repo
-
-agtesting:
-  file.absent:
-    - name: /etc/yum.repos.d/epel-testing.repo
-
-agssrepo:
-  file.absent:
-    - name: /etc/yum.repos.d/saltstack.repo
-
-agwazrepo:
-  file.absent:
-    - name: /etc/yum.repos.d/wazuh.repo
-
-{% else %}
-
-{{sls}}_state_not_allowed:
-  test.fail_without_changes:
-    - name: {{sls}}_state_not_allowed
-
-{% endif %}

salt/allowed_states.map.jinja

@@ -45,7 +45,8 @@
           'schedule',
           'soctopus',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-heavynode': [
           'ca',
@@ -108,7 +109,8 @@
           'zeek',
           'schedule',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-manager': [
           'salt.master',
@@ -127,7 +129,8 @@
           'utility',
           'schedule',
           'soctopus',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-managersearch': [
           'salt.master',
@@ -146,7 +149,8 @@
           'utility',
           'schedule',
           'soctopus',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-node': [
           'ca',
@@ -178,7 +182,8 @@
           'schedule',
           'soctopus',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-sensor': [
           'ca',
@@ -237,7 +242,7 @@
     {% do allowed_states.append('kibana') %}
   {% endif %}
 
-  {% if CURATOR and grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+  {% if grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-manager'] %}
     {% do allowed_states.append('curator') %}
   {% endif %}
 
@@ -296,4 +301,4 @@
 {% endif %}
 
 {# all nodes can always run salt.minion state #}
-{% do allowed_states.append('salt.minion') %}
+{% do allowed_states.append('salt.minion') %}

salt/ca/init.sls

@@ -43,8 +43,9 @@ pki_private_key:
     - require:
       - file: /etc/pki
     - timeout: 30
-    - retry: 5
-    - interval: 30
+    - retry:
+        attempts: 5
+        interval: 30
 
 x509_pem_entries:
   module.run:

salt/common/files/99-reserved-ports.conf

@@ -1 +1 @@
-net.ipv4.ip_local_reserved_ports="55000,57314"
+net.ipv4.ip_local_reserved_ports=55000,57314,47760-47860

salt/common/files/log-rotate.conf

@@ -22,6 +22,7 @@
 /opt/so/log/salt/so-salt-minion-check
 /opt/so/log/salt/minion
 /opt/so/log/salt/master
+/opt/so/log/logscan/*.log
 {
     {{ logrotate_conf | indent(width=4) }}
 }

salt/common/files/soversion

@@ -0,0 +1,2 @@
+{%- set VERSION = salt['pillar.get']('global:soversion') -%}
+{{ VERSION }}

salt/common/files/vimrc

@@ -0,0 +1,6 @@
+" Activates filetype detection
+filetype plugin indent on
+
+" Sets .sls files to use YAML syntax highlighting
+autocmd BufNewFile,BufRead *.sls set syntax=yaml
+set number

salt/common/init.sls

@@ -2,6 +2,7 @@
 {% if sls in allowed_states %}
 
 {% set role = grains.id.split('_') | last %}
+{% from 'elasticsearch/auth.map.jinja' import ELASTICAUTH with context %}
 
 # Remove variables.txt from /tmp - This is temp
 rmvariablesfile:
@@ -49,6 +50,11 @@ sosaltstackperms:
     - gid: 939
     - dir_mode: 770
 
+so_log_perms:
+  file.directory:
+    - name: /opt/so/log
+    - dir_mode: 755
+
 # Create a state directory
 statedir:
   file.directory:
@@ -64,20 +70,12 @@ salttmp:
     - group: 939
     - makedirs: True
 
-# Install epel
-{% if grains['os'] == 'CentOS' %}
-repair_yumdb:
-  cmd.run:
-    - name: 'mv -f /var/lib/rpm/__db* /tmp && yum clean all'
-    - onlyif:
-      - 'yum check-update 2>&1 | grep "Error: rpmdb open failed"'
-
-epel:
-  pkg.installed:
-    - skip_suggestions: True
-    - pkgs:
-      - epel-release
-{% endif %}
+# VIM config
+vimconfig:
+  file.managed:
+    - name: /root/.vimrc
+    - source: salt://common/files/vimrc
+    - replace: False
 
 # Install common packages
 {% if grains['os'] != 'CentOS' %}     
@@ -90,7 +88,6 @@ commonpkgs:
       - ntpdate
       - jq
       - python3-docker
-      - docker-ce
       - curl
       - ca-certificates
       - software-properties-common
@@ -99,17 +96,21 @@ commonpkgs:
       - netcat
       - python3-mysqldb
       - sqlite3
-      - argon2
       - libssl-dev
       - python3-dateutil
       - python3-m2crypto
       - python3-mysqldb
+      - python3-packaging
       - git
+      - vim
+
 heldpackages:
   pkg.installed:
     - pkgs:
-      - containerd.io: 1.2.13-2
-      - docker-ce: 5:19.03.14~3-0~ubuntu-bionic
+      - containerd.io: 1.4.4-1
+      - docker-ce: 5:20.10.5~3-0~ubuntu-bionic
+      - docker-ce-cli: 5:20.10.5~3-0~ubuntu-bionic
+      - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-bionic
     - hold: True
     - update_holds: True
 
@@ -127,7 +128,6 @@ commonpkgs:
       - net-tools
       - curl
       - sqlite
-      - argon2
       - mariadb-devel
       - nmap-ncat
       - python3
@@ -135,17 +135,21 @@ commonpkgs:
       - python36-dateutil
       - python36-m2crypto
       - python36-mysql
+      - python36-packaging
       - yum-utils
       - device-mapper-persistent-data
       - lvm2
       - openssl
       - git
+      - vim-enhanced
 
 heldpackages:
   pkg.installed:
     - pkgs:
-      - containerd.io: 1.2.13-3.2.el7
-      - docker-ce: 3:19.03.14-3.el7
+      - containerd.io: 1.4.4-3.1.el7
+      - docker-ce: 3:20.10.5-3.el7
+      - docker-ce-cli: 1:20.10.5-3.el7
+      - docker-ce-rootless-extras: 20.10.5-3.el7
     - hold: True
     - update_holds: True
 {% endif %}
@@ -164,6 +168,14 @@ alwaysupdated:
 Etc/UTC:
   timezone.system
 
+elastic_curl_config:
+  file.managed:
+    - name: /opt/so/conf/elasticsearch/curl.config
+    - source: salt://elasticsearch/curl.config
+    - mode: 600
+    - show_changes: False
+    - makedirs: True
+
 # Sync some Utilities
 utilsyncscripts:
   file.recurse:
@@ -173,6 +185,10 @@ utilsyncscripts:
     - file_mode: 755
     - template: jinja
     - source: salt://common/tools/sbin
+    - defaults:
+        ELASTICCURL: 'curl'
+    - context:
+        ELASTICCURL: {{ ELASTICAUTH.elasticcurl }}
 
 {% if role in ['eval', 'standalone', 'sensor', 'heavynode'] %}
 # Add sensor cleanup
@@ -230,6 +246,30 @@ commonlogrotateconf:
     - month: '*'
     - dayweek: '*'
 
+# Create the status directory
+sostatusdir:
+  file.directory:
+    - name: /opt/so/log/sostatus
+    - user: 0
+    - group: 0
+    - makedirs: True
+
+sostatus_log:
+  file.managed:
+    - name: /opt/so/log/sostatus/status.log
+    - mode: 644
+    
+# Install sostatus check cron
+'/usr/sbin/so-status -q; echo $? > /opt/so/log/sostatus/status.log 2>&1':
+  cron.present:
+    - user: root
+    - minute: '*/1'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
+
 {% if role in ['eval', 'manager', 'managersearch', 'standalone'] %}
 # Lock permissions on the backup directory
 backupdir:
@@ -249,6 +289,14 @@ backupdir:
     - daymonth: '*'
     - month: '*'
     - dayweek: '*'
+{% else %}
+soversionfile:
+  file.managed:
+    - name: /etc/soversion
+    - source: salt://common/files/soversion
+    - mode: 644
+    - template: jinja
+    
 {% endif %}
 
 # Manager daemon.json
@@ -266,9 +314,10 @@ docker:
       - file: docker_daemon
 
 # Reserve OS ports for Docker proxy in case boot settings are not already applied/present
+# 55000 = Wazuh, 57314 = Strelka, 47760-47860 = Zeek
 dockerapplyports:
     cmd.run:
-      - name: if [ ! -s /etc/sysctl.d/99-reserved-ports.conf ]; then sysctl -w net.ipv4.ip_local_reserved_ports="55000,57314"; fi
+      - name: if [ ! -s /etc/sysctl.d/99-reserved-ports.conf ]; then sysctl -w net.ipv4.ip_local_reserved_ports="55000,57314,47760-47860"; fi
 
 # Reserve OS ports for Docker proxy
 dockerreserveports:
@@ -277,6 +326,16 @@ dockerreserveports:
     - name: /etc/sysctl.d/99-reserved-ports.conf
 
 {% if salt['grains.get']('sosmodel', '') %}
+  {% if grains['os'] == 'CentOS' %}     
+# Install Raid tools
+raidpkgs:
+  pkg.installed:
+    - skip_suggestions: True
+    - pkgs:
+      - securityonion-raidtools
+      - securityonion-megactl
+  {% endif %}
+
 # Install raid check cron
 /usr/sbin/so-raid-status > /dev/null 2>&1:
   cron.present:

salt/common/tools/sbin/so-checkin

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-salt-call state.highstate
+salt-call state.highstate -linfo 

salt/common/tools/sbin/so-common

@@ -15,6 +15,8 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+DEFAULT_SALT_DIR=/opt/so/saltstack/default
+
 # Check for prerequisites
 if [ "$(id -u)" -ne 0 ]; then
 	echo "This script must be run using sudo!"
@@ -97,6 +99,60 @@ check_password() {
 	return $?
 }
 
+check_elastic_license() {
+
+	[ -n "$TESTING" ] && return
+
+	# See if the user has already accepted the license
+	if [ ! -f /opt/so/state/yeselastic.txt ]; then
+	  elastic_license
+	else
+	  echo "Elastic License has already been accepted"
+	fi  
+}
+
+copy_new_files() {
+  # Copy new files over to the salt dir
+  cd $UPDATE_DIR
+  rsync -a salt $DEFAULT_SALT_DIR/
+  rsync -a pillar $DEFAULT_SALT_DIR/
+  chown -R socore:socore $DEFAULT_SALT_DIR/
+  chmod 755 $DEFAULT_SALT_DIR/pillar/firewall/addfirewall.sh
+  cd /tmp
+}
+
+disable_fastestmirror() {
+	sed -i 's/enabled=1/enabled=0/' /etc/yum/pluginconf.d/fastestmirror.conf
+}
+
+elastic_license() {
+
+read -r -d '' message <<- EOM
+\n
+Starting in Elastic Stack version 7.11, the Elastic Stack binaries are only available under the Elastic License:
+https://securityonion.net/elastic-license
+
+Please review the Elastic License:
+https://www.elastic.co/licensing/elastic-license
+
+Do you agree to the terms of the Elastic License?
+
+If so, type AGREE to accept the Elastic License and continue.  Otherwise, press Enter to exit this program without making any changes.
+EOM
+
+	AGREED=$(whiptail --title "$whiptail_title" --inputbox \
+	"$message" 20 75 3>&1 1>&2 2>&3)
+
+	if [ "${AGREED^^}" = 'AGREE' ]; then
+		mkdir -p /opt/so/state
+		touch /opt/so/state/yeselastic.txt 
+	else
+		echo "Starting in 2.3.40 you must accept the Elastic license if you want to run Security Onion."
+		exit 1
+	fi
+
+}
+
 fail() {
 	msg=$1
 	echo "ERROR: $msg"
@@ -109,6 +165,23 @@ get_random_value() {
 	head -c 5000 /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w $length | head -n 1
 }
 
+gpg_rpm_import() {
+	if [[ "$OS" == "centos" ]]; then
+	    if [[ "$WHATWOULDYOUSAYYAHDOHERE" == "setup" ]]; then
+	        local RPMKEYSLOC="../salt/repo/client/files/centos/keys"
+	    else
+	        local RPMKEYSLOC="$UPDATE_DIR/salt/repo/client/files/centos/keys"
+	    fi
+	
+	    RPMKEYS=('RPM-GPG-KEY-EPEL-7' 'GPG-KEY-WAZUH' 'docker.pub' 'SALTSTACK-GPG-KEY.pub' 'securityonion.pub')
+
+	    for RPMKEY in "${RPMKEYS[@]}"; do
+    	        rpm --import $RPMKEYSLOC/$RPMKEY
+	        echo "Imported $RPMKEY"
+	    done
+	fi
+}
+
 header() {
 	printf '%s\n' "" "$banner" " $*" "$banner" 
 }
@@ -166,6 +239,7 @@ lookup_salt_value() {
 	key=$1
 	group=$2
 	kind=$3
+	output=${4:-newline_values_only}
 
 	if [ -z "$kind" ]; then
 		kind=pillar
@@ -175,7 +249,7 @@ lookup_salt_value() {
 		group=${group}:
 	fi
 
-	salt-call --no-color  ${kind}.get ${group}${key} --out=newline_values_only
+	salt-call --no-color  ${kind}.get ${group}${key} --out=${output}
 }
 
 lookup_pillar() {
@@ -203,7 +277,7 @@ lookup_role() {
 
 require_manager() {
 	if is_manager_node; then
-		echo "This is a manager, We can proceed."
+		echo "This is a manager, so we can proceed."
 	else
 		echo "Please run this command on the manager; the manager controls the grid."
 		exit 1
@@ -216,6 +290,7 @@ retry() {
 	cmd=$3
 	expectedOutput=$4
 	attempt=0
+	local exitcode=0
 	while [[ $attempt -lt $maxAttempts ]]; do			
 		attempt=$((attempt+1))
 		echo "Executing command with retry support: $cmd"
@@ -235,7 +310,29 @@ retry() {
 		sleep $sleepDelay
 	done
 	echo "Command continues to fail; giving up."
-	return 1
+	return $exitcode
+}
+
+run_check_net_err() {
+	local cmd=$1
+	local err_msg=${2:-"Unknown error occured, please check /root/$WHATWOULDYOUSAYYAHDOHERE.log for details."} # Really need to rename that variable
+	local no_retry=$3
+
+	local exit_code
+	if [[ -z $no_retry ]]; then
+		retry 5 60 "$cmd"
+		exit_code=$?
+	else
+		eval "$cmd"
+		exit_code=$?
+	fi
+	
+	if [[ $exit_code -ne 0 ]]; then
+		ERR_HANDLED=true
+		[[ -z $no_retry ]] || echo "Command failed with error $exit_code"
+		echo "$err_msg"
+		exit $exit_code
+	fi
 }
 
 set_os() {
@@ -250,6 +347,12 @@ set_minionid() {
 	MINIONID=$(lookup_grain id)
 }
 
+set_palette() {
+	if [ "$OS" == ubuntu ]; then
+	    update-alternatives --set newt-palette /etc/newt/palette.original
+    fi
+}
+
 set_version() {
 	CURRENTVERSION=0.0.0
 	if [ -f /etc/soversion ]; then
@@ -269,6 +372,14 @@ set_version() {
 	fi
 }
 
+has_uppercase() {
+	local string=$1
+
+	echo "$string" | grep -qP '[A-Z]' \
+		&& return 0 \
+		|| return 1
+}
+
 valid_cidr() {
 	# Verify there is a backslash in the string
 	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
@@ -340,6 +451,40 @@ valid_int() {
 
 # {% raw %}
 
+valid_proxy() {
+	local proxy=$1
+	local url_prefixes=( 'http://' 'https://' )
+
+	local has_prefix=false
+	for prefix in "${url_prefixes[@]}"; do
+		echo "$proxy" | grep -q "$prefix" && has_prefix=true && proxy=${proxy#"$prefix"} && break
+	done
+	
+	local url_arr
+	mapfile -t url_arr <<< "$(echo "$proxy" | tr ":" "\n")"
+
+	local valid_url=true
+	if ! valid_ip4 "${url_arr[0]}" && ! valid_fqdn "${url_arr[0]}" && ! valid_hostname "${url_arr[0]}"; then
+		valid_url=false
+	fi
+
+	[[ $has_prefix == true ]] && [[ $valid_url == true ]] && return 0 || return 1
+}
+
+valid_ntp_list() {
+	local string=$1
+	local ntp_arr
+	IFS="," read -r -a ntp_arr <<< "$string"
+
+	for ntp in "${ntp_arr[@]}"; do
+		if ! valid_ip4 "$ntp" && ! valid_hostname "$ntp" && ! valid_fqdn "$ntp"; then
+			return 1
+		fi
+	done
+
+	return 0
+}
+
 valid_string() {
 	local str=$1
 	local min_length=${2:-1}
@@ -360,12 +505,14 @@ wait_for_web_response() {
 	url=$1
 	expected=$2
 	maxAttempts=${3:-300}
+	curlcmd=${4:-curl}
 	logfile=/root/wait_for_web_response.log
+	truncate -s 0 "$logfile"
 	attempt=0
 	while [[ $attempt -lt $maxAttempts ]]; do
 		attempt=$((attempt+1))
 		echo "Waiting for value '$expected' at '$url' ($attempt/$maxAttempts)"
-		result=$(curl -ks -L $url)
+		result=$($curlcmd -ks -L $url)
 		exitcode=$?
 
 		echo "--------------------------------------------------" >> $logfile

salt/common/tools/sbin/so-config-backup

@@ -35,6 +35,7 @@ if [ ! -f $BACKUPFILE ]; then
   {%- endfor %}
   tar -rf $BACKUPFILE /etc/pki
   tar -rf $BACKUPFILE /etc/salt
+  tar -rf $BACKUPFILE /opt/so/conf/kratos
 
 fi
 

salt/common/tools/sbin/so-cortex-user-add

@@ -30,7 +30,7 @@ fi
 
 USER=$1
 
-CORTEX_KEY=$(lookup_pillar cortexkey)
+CORTEX_KEY=$(lookup_pillar cortexorguserkey)
 CORTEX_API_URL="$(lookup_pillar url_base)/cortex/api"
 CORTEX_ORG_NAME=$(lookup_pillar cortexorgname)
 CORTEX_USER=$USER

salt/common/tools/sbin/so-cortex-user-enable

@@ -30,7 +30,7 @@ fi
 
 USER=$1
 
-CORTEX_KEY=$(lookup_pillar cortexkey)
+CORTEX_KEY=$(lookup_pillar cortexorguserkey)
 CORTEX_API_URL="$(lookup_pillar url_base)/cortex/api"
 CORTEX_USER=$USER
 

salt/common/tools/sbin/so-docker-prune

@@ -0,0 +1,102 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import sys, argparse, re, docker
+from packaging.version import Version, InvalidVersion
+from itertools import groupby, chain
+
+
+def get_image_name(string) -> str:
+  return ':'.join(string.split(':')[:-1])
+
+
+def get_so_image_basename(string) -> str:
+  return get_image_name(string).split('/so-')[-1]
+
+
+def get_image_version(string) -> str:
+  ver = string.split(':')[-1]
+  if ver == 'latest':
+    # Version doesn't like "latest", so use a high semver
+    return '99999.9.9'
+  else:
+    try:
+      Version(ver)
+    except InvalidVersion:
+      # Also return a very high semver for any version 
+      # with a dash in it since it will likely be a dev version of some kind
+      if '-' in ver:
+        return '999999.9.9'
+    return ver
+
+
+def main(quiet):
+  client = docker.from_env()
+
+  # Prune old/stopped containers
+  if not quiet: print('Pruning old containers')
+  client.containers.prune()
+
+  image_list = client.images.list(filters={ 'dangling': False })
+
+  # Map list of image objects to flattened list of tags (format: "name:version")
+  tag_list = list(chain.from_iterable(list(map(lambda x: x.attrs.get('RepoTags'), image_list))))
+
+  # Filter to only SO images (base name begins with "so-")
+  tag_list = list(filter(lambda x: re.match(r'^.*\/so-[^\/]*$', get_image_name(x)), tag_list))
+
+  # Group tags into lists by base name (sort by same projection first)
+  tag_list.sort(key=lambda x: get_so_image_basename(x))
+  grouped_tag_lists = [ list(it) for _, it in groupby(tag_list, lambda x: get_so_image_basename(x)) ]
+
+  no_prunable = True
+  for t_list in grouped_tag_lists:
+    try:
+      # Group tags by version, in case multiple images exist with the same version string
+      t_list.sort(key=lambda x: Version(get_image_version(x)), reverse=True)
+      grouped_t_list = [ list(it) for _,it in groupby(t_list, lambda x: get_image_version(x)) ]
+
+      # Keep the 2 most current version groups
+      if len(grouped_t_list) <= 2:
+        continue
+      else:
+        no_prunable = False
+        for group in grouped_t_list[2:]:
+          for tag in group:
+            if not quiet: print(f'Removing image {tag}')
+            try:
+              client.images.remove(tag, force=True)
+            except docker.errors.ClientError as e:
+              print(f'Could not remove image {tag}, continuing...')
+    except (docker.errors.APIError, InvalidVersion) as e:
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+    except Exception as e:
+      print('Unhandled exception occurred:')
+      print(f'so-{get_so_image_basename(t_list[0])}: {e}', file=sys.stderr)
+      exit(1)
+
+  if no_prunable and not quiet:
+    print('No Security Onion images to prune')
+
+
+if __name__ == "__main__":
+  main_parser = argparse.ArgumentParser(add_help=False)
+  main_parser.add_argument('-q', '--quiet', action='store_const', const=True, required=False)
+  args = main_parser.parse_args(sys.argv[1:])
+ 
+  main(args.quiet)

salt/common/tools/sbin/so-elastalert-create

@@ -145,9 +145,9 @@ EOF
    rulename=$(echo ${raw_rulename,,} | sed 's/ /_/g')
 
 cat << EOF >> "$rulename.yaml"
-# Elasticsearch Host
-es_host: elasticsearch
-es_port: 9200
+# Elasticsearch Host Override (optional)
+# es_host: elasticsearch
+# es_port: 9200
 
 # (Required)
 # Rule name, must be unique

salt/common/tools/sbin/so-elastic-auth

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if [ -f "/usr/sbin/so-common" ]; then
+  . /usr/sbin/so-common
+fi
+
+ES_AUTH_PILLAR=${ELASTIC_AUTH_PILLAR:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+ES_USERS_FILE=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+
+authEnable=$1
+
+if ! grep -q "enabled: " "$ES_AUTH_PILLAR"; then
+  echo "Elastic auth pillar file is invalid. Unable to proceed."
+  exit 1
+fi
+
+function restart() {
+  if [[ -z "$ELASTIC_AUTH_SKIP_HIGHSTATE" ]]; then
+    echo "Elasticsearch on all affected minions will now be stopped and then restarted..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' cmd.run so-elastic-stop queue=True
+    echo "Applying highstate to all affected minions..."
+    salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.highstate queue=True
+  fi
+}
+
+if [[ "$authEnable" == "true" ]]; then
+  if grep -q "enabled: False" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: False/enabled: True/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now enabled."
+    if grep -q "argon" "$ES_USERS_FILE"; then
+      echo ""
+      echo "IMPORTANT: The following users will need to change their password, after logging into SOC, in order to access Kibana:"
+      grep argon "$ES_USERS_FILE" | cut -d ":" -f 1
+    fi
+  else
+    echo "Auth is already enabled."
+  fi
+elif [[ "$authEnable" == "false" ]]; then
+  if grep -q "enabled: True" "$ES_AUTH_PILLAR"; then
+    sed -i 's/enabled: True/enabled: False/g' "$ES_AUTH_PILLAR"
+    restart
+    echo "Elastic auth is now disabled."
+  else
+    echo "Auth is already disabled."
+  fi
+else
+  echo "Usage: $0 <true|false>"
+  echo ""
+  echo "Toggles Elastic authentication. Elasticsearch will be restarted on each affected minion."
+  echo ""
+fi

salt/common/tools/sbin/so-elastic-clear

@@ -50,11 +50,7 @@ done
 if [ $SKIP -ne 1 ]; then
         # List indices
         echo 
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -k -L https://{{ NODEIP }}:9200/_cat/indices?v
-        {% else %}
-        curl -L {{ NODEIP }}:9200/_cat/indices?v
-        {% endif %}
+        {{ ELASTICCURL }} -k -L https://{{ NODEIP }}:9200/_cat/indices?v
         echo
         # Inform user we are about to delete all data
         echo
@@ -93,18 +89,10 @@ fi
 # Delete data
 echo "Deleting data..."
 
-{% if grains['role'] in ['so-node','so-heavynode'] %}
-INDXS=$(curl -s -XGET -k -L https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
-{% else %}
-INDXS=$(curl -s -XGET -L {{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
-{% endif %}
+INDXS=$({{ ELASTICCURL }} -s -XGET -k -L https://{{ NODEIP }}:9200/_cat/indices?v | egrep 'logstash|elastalert|so-' | awk '{ print $3 }')
 for INDX in ${INDXS}
 do
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -XDELETE -k -L https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
-    {% else %}
-    curl -XDELETE -L "{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
-    {% endif %}
+    {{ ELASTICCURL }} -XDELETE -k -L https://"{{ NODEIP }}:9200/${INDX}" > /dev/null 2>&1
 done
 
 #Start Logstash/Filebeat

salt/common/tools/sbin/so-elasticsearch-indices-list

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/indices?pretty

salt/common/tools/sbin/so-elasticsearch-indices-rw

@@ -21,6 +21,5 @@ THEHIVEESPORT=9400
 
 echo "Removing read only attributes for indices..."
 echo
-for p in $ESPORT $THEHIVEESPORT; do
-   curl -XPUT -H "Content-Type: application/json" -L http://$IP:$p/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
-done
+{{ ELASTICCURL }} -s -k -XPUT -H "Content-Type: application/json" -L https://$IP:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;
+{{ ELASTICCURL }} -XPUT -H "Content-Type: application/json" -L http://$IP:9400/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}' 2>&1 | if grep -q ack; then echo "Index settings updated..."; else echo "There was any issue updating the read-only attribute.  Please ensure Elasticsearch is running.";fi;

salt/common/tools/sbin/so-elasticsearch-pipeline-stats

@@ -19,15 +19,7 @@
 . /usr/sbin/so-common
 
 if [ "$1" == "" ]; then
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
-        {% else %}
-        curl -s -L {{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
-        {% endif %}
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines"
 else
-        {% if grains['role'] in ['so-node','so-heavynode'] %}
-        curl -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
-        {% else %}
-        curl -s -L {{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
-        {% endif %}
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_nodes/stats | jq .nodes | jq ".[] | .ingest.pipelines.\"$1\""
 fi

salt/common/tools/sbin/so-elasticsearch-pipeline-view

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq .
+else
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq .[]
+fi

salt/common/tools/sbin/so-elasticsearch-pipelines-list

@@ -17,15 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
-    {% else %}
-    curl -s -L {{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/* | jq 'keys'
 else
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
-    {% else %}
-    curl -s -L {{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_ingest/pipeline/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-query

@@ -0,0 +1,37 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+. /usr/sbin/so-common
+
+if [[ $# -lt 1 ]]; then
+	echo "Submit a cURL request to the local Security Onion Elasticsearch host."
+	echo ""
+	echo "Usage: $0 <PATH> [ARGS,...]"
+	echo ""
+  echo "Where "
+  echo "   PATH represents the elastic function being requested."
+  echo "   ARGS is used to specify additional, optional curl parameters."
+  echo ""
+  echo "Examples:"
+  echo "   $0 /"
+  echo "   $0 '*:so-*/_search' -d '{\"query\": {\"match_all\": {}},\"size\": 1}' | jq"
+  exit 1
+fi
+
+QUERYPATH=$1
+shift
+
+{{ ELASTICCURL }} -s -k -L -H "Content-Type: application/json" "https://localhost:9200/${QUERYPATH}" "$@"

salt/common/tools/sbin/so-elasticsearch-shards-list

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_cat/shards?pretty

salt/common/tools/sbin/so-elasticsearch-template-remove

@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+{{ ELASTICCURL }} -s -k -L -XDELETE https://{{ NODEIP }}:9200/_template/$1

salt/common/tools/sbin/so-elasticsearch-template-view

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/* | jq .
+else
+        {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq .
+fi

salt/common/tools/sbin/so-elasticsearch-templates-list

@@ -17,15 +17,7 @@
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 . /usr/sbin/so-common
 if [ "$1" == "" ]; then
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k -L https://{{ NODEIP }}:9200/_template/* | jq 'keys'
-    {% else %}
-    curl -s -L {{ NODEIP }}:9200/_template/* | jq 'keys'
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/* | jq 'keys'
 else
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq
-    {% else %}
-    curl -s -L {{ NODEIP }}:9200/_template/$1 | jq
-    {% endif %}
+    {{ ELASTICCURL }} -s -k -L https://{{ NODEIP }}:9200/_template/$1 | jq
 fi

salt/common/tools/sbin/so-elasticsearch-templates-load

@@ -1,8 +1,5 @@
-{%- set mainint = salt['pillar.get']('host:mainint') %}
-{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
-
 #!/bin/bash
-# Copyright 2014,2015,2016,2017,2018,2019 Security Onion Solutions, LLC
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,6 +14,9 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
 default_conf_dir=/opt/so/conf
 ELASTICSEARCH_HOST="{{ MYIP }}"
 ELASTICSEARCH_PORT=9200
@@ -30,11 +30,7 @@ echo -n "Waiting for ElasticSearch..."
 COUNT=0
 ELASTICSEARCH_CONNECTED="no"
 while [[ "$COUNT" -le 240 ]]; do
-    {% if grains['role'] in ['so-node','so-heavynode'] %}
-    curl -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
-    {% else %}
-	curl --output /dev/null --silent --head --fail -L http://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
-	{% endif %}
+    {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
 	if [ $? -eq 0 ]; then
 		ELASTICSEARCH_CONNECTED="yes"
 		echo "connected!"
@@ -55,11 +51,7 @@ cd ${ELASTICSEARCH_TEMPLATES}
 
 
 echo "Loading templates..."
-{% if grains['role'] in ['so-node','so-heavynode'] %}
-for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; curl -k ${ELASTICSEARCH_AUTH} -s -XPUT -L https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
-{% else %}
-for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; curl ${ELASTICSEARCH_AUTH} -s -XPUT -L http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
-{% endif %}
+for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; {{ ELASTICCURL }} -k ${ELASTICSEARCH_AUTH} -s -XPUT -L https://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
 echo
 
 cd - >/dev/null

salt/common/tools/sbin/so-elasticsearch-wait

@@ -0,0 +1,5 @@
+#!/bin/bash
+
+. /usr/sbin/so-common
+
+wait_for_web_response "https://localhost:9200/_cat/indices/.kibana*" "green open" 300 "{{ ELASTICCURL }}"

salt/common/tools/sbin/so-features-enable

@@ -1,53 +0,0 @@
-#!/bin/bash
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
-
-#    This program is free software: you can redistribute it and/or modify
-#    it under the terms of the GNU General Public License as published by
-#    the Free Software Foundation, either version 3 of the License, or
-#    (at your option) any later version.
-#
-#    This program is distributed in the hope that it will be useful,
-#    but WITHOUT ANY WARRANTY; without even the implied warranty of
-#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#    GNU General Public License for more details.
-#
-#    You should have received a copy of the GNU General Public License
-#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-. /usr/sbin/so-common
-. /usr/sbin/so-image-common
-local_salt_dir=/opt/so/saltstack/local
-
-cat << EOF
-This program will switch from the open source version of the Elastic Stack to the Features version licensed under the Elastic license.
-If you proceed, then we will download new Docker images and restart services.
-
-Please review the Elastic license:
-https://raw.githubusercontent.com/elastic/elasticsearch/master/licenses/ELASTIC-LICENSE.txt
-
-Please also note that, if you have a distributed deployment and continue with this change, Elastic traffic between nodes will change from encrypted to cleartext!
-(We expect to support Elastic Features Security at some point in the future.)
-
-Do you agree to the terms of the Elastic license and understand the note about encryption?
-
-If so, type AGREE to accept the Elastic license and continue.  Otherwise, just press Enter to exit this program without making any changes.
-EOF
-
-read INPUT
-if [ "$INPUT" != "AGREE" ]; then
-        exit
-fi
-
-echo "Please wait while switching to Elastic Features."
-
-require_manager
-
-TRUSTED_CONTAINERS=( \
-  "so-elasticsearch" \
-  "so-filebeat" \
-  "so-kibana" \
-  "so-logstash" )
-update_docker_containers "features" "-features"
-
-# Modify global.sls to enable Features
-sed -i 's/features: False/features: True/' $local_salt_dir/pillar/global.sls

salt/common/tools/sbin/so-filebeat-module-setup

@@ -0,0 +1,67 @@
+#!/bin/bash
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
+default_conf_dir=/opt/so/conf
+ELASTICSEARCH_HOST="{{ MYIP }}"
+ELASTICSEARCH_PORT=9200
+#ELASTICSEARCH_AUTH=""
+
+# Define a default directory to load pipelines from
+FB_MODULE_YML="/usr/share/filebeat/module-setup.yml"
+
+
+# Wait for ElasticSearch to initialize
+echo -n "Waiting for ElasticSearch..."
+COUNT=0
+ELASTICSEARCH_CONNECTED="no"
+while [[ "$COUNT" -le 240 ]]; do
+        {{ ELASTICCURL }} -k --output /dev/null --silent --head --fail -L https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+        if [ $? -eq 0 ]; then
+                ELASTICSEARCH_CONNECTED="yes"
+                echo "connected!"
+                break
+        else
+                ((COUNT+=1))
+                sleep 1
+                echo -n "."
+        fi
+done
+if [ "$ELASTICSEARCH_CONNECTED" == "no" ]; then
+        echo
+        echo -e "Connection attempt timed out.  Unable to connect to ElasticSearch.  \nPlease try: \n  -checking log(s) in /var/log/elasticsearch/\n  -running 'sudo docker ps' \n  -running 'sudo so-elastic-restart'"
+        echo
+fi
+echo "Testing to see if the pipelines are already applied"
+ESVER=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT" |jq .version.number |tr -d \")
+PIPELINES=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"/_ingest/pipeline/filebeat-$ESVER-suricata-eve-pipeline | jq . | wc -c)
+
+if [[ "$PIPELINES" -lt 5 ]]; then
+  echo "Setting up ingest pipeline(s)"
+
+  for MODULE in activemq apache auditd aws azure barracuda bluecoat cef checkpoint cisco coredns crowdstrike cyberark cylance elasticsearch envoyproxy f5 fortinet gcp google_workspace googlecloud gsuite haproxy ibmmq icinga iis imperva infoblox iptables juniper kafka kibana logstash microsoft misp mongodb mssql mysql nats netscout nginx o365 okta osquery panw postgresql rabbitmq radware redis santa snort snyk sonicwall sophos squid suricata system tomcat traefik zeek zscaler
+  do
+    echo "Loading $MODULE"
+    docker exec -i so-filebeat filebeat setup modules -pipelines -modules $MODULE -c $FB_MODULE_YML
+    sleep 2
+  done
+else
+  exit 0
+fi
+
+

salt/common/tools/sbin/so-firewall

@@ -35,6 +35,7 @@ def showUsage(options, args):
   print('')
   print('  General commands:')
   print('    help           - Prints this usage information.')
+  print('    apply          - Apply the firewall state.')
   print('')
   print('  Host commands:')
   print('    listhostgroups - Lists the known host groups.')
@@ -66,7 +67,7 @@ def checkDefaultPortsOption(options):
 
 def checkApplyOption(options):
   if "--apply" in options:
-    return apply()
+    return apply(None, None)
 
 def loadYaml(filename):
   file = open(filename, "r")
@@ -328,7 +329,7 @@ def removehost(options, args):
     code = checkApplyOption(options)
   return code
 
-def apply():
+def apply(options, args):
   proc = subprocess.run(['salt-call', 'state.apply', 'firewall', 'queue=True'])
   return proc.returncode
 
@@ -356,7 +357,8 @@ def main():
     "addport": addport,
     "removeport": removeport,
     "addhostgroup": addhostgroup,
-    "addportgroup": addportgroup
+    "addportgroup": addportgroup,
+    "apply": apply
   }
 
   code=1

salt/common/tools/sbin/so-fleet-user-update

@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name>"
+    echo ""
+    echo "Update password for an existing Fleet user. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+MYSQL_PASS=$(lookup_pillar_secret mysql)
+FLEET_IP=$(lookup_pillar fleet_ip)
+FLEET_USER=$USER
+
+# test existence of user
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "SELECT count(1) FROM users WHERE username='$FLEET_USER'" 2>/dev/null | tail -1)
+if [[ $? -ne 0 ]] || [[ $MYSQL_OUTPUT -ne 1 ]] ; then
+    echo "Test for username [${FLEET_USER}] failed"
+    echo "    expect 1 hit in users database, return $MYSQL_OUTPUT hit(s)."
+    echo "Unable to update Fleet user password."
+    exit 2
+fi
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -rs FLEET_PASS
+
+if ! check_password "$FLEET_PASS"; then
+  echo "Password is invalid. Please exclude single quotes, double quotes and backslashes from the password."
+  exit 2
+fi
+
+FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_PASS'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
+if [[ $? -ne 0 ]]; then
+	echo "Failed to generate Fleet password hash"
+	exit 2
+fi
+
+
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "UPDATE users SET password='$FLEET_HASH', salt='' where username='$FLEET_USER'" 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully updated Fleet user password"
+else
+    echo "Unable to update Fleet user password"
+    echo "$MYSQL_OUTPUT"
+    exit 2
+fi

salt/common/tools/sbin/so-grafana-dashboard-folder-delete

@@ -0,0 +1,17 @@
+# this script is used to delete the default Grafana dashboard folders that existed prior to Grafana dashboard and Salt management changes in 2.3.70
+
+folders=$(curl -X GET http://admin:{{salt['pillar.get']('secrets:grafana_admin')}}@localhost:3000/api/folders | jq -r '.[] | @base64')
+delfolder=("Manager" "Manager Search" "Sensor Nodes" "Search Nodes" "Standalone" "Eval Mode")
+
+for row in $folders; do
+   title=$(echo ${row} | base64 --decode | jq -r '.title')
+   uid=$(echo ${row} | base64 --decode | jq -r '.uid')
+
+  if [[ " ${delfolder[@]} " =~ " ${title} " ]]; then
+   curl -X DELETE http://admin:{{salt['pillar.get']('secrets:grafana_admin')}}@localhost:3000/api/folders/$uid
+  fi
+done
+
+echo "so-grafana-dashboard-folder-delete has been run to delete default Grafana dashboard folders that existed prior to 2.3.70" > /opt/so/state/so-grafana-dashboard-folder-delete-complete
+
+exit 0

salt/common/tools/sbin/so-image-common

@@ -18,6 +18,7 @@
 # NOTE: This script depends on so-common
 IMAGEREPO=security-onion-solutions
 
+# shellcheck disable=SC2120
 container_list() {
   MANAGERCHECK=$1
   
@@ -47,20 +48,17 @@ container_list() {
     TRUSTED_CONTAINERS=(
       "so-acng"
       "so-curator"
-      "so-domainstats"
       "so-elastalert"
       "so-elasticsearch"
       "so-filebeat"
       "so-fleet"
       "so-fleet-launcher"
-      "so-freqserver"
       "so-grafana"
       "so-idstools"
       "so-influxdb"
       "so-kibana"
       "so-kratos"
       "so-logstash"
-      "so-minio"
       "so-mysql"
       "so-nginx"
       "so-pcaptools"
@@ -131,13 +129,13 @@ update_docker_containers() {
   mkdir -p $SIGNPATH >> "$LOG_FILE" 2>&1 
 
   # Let's make sure we have the public key
-  retry 50 10 "curl -sSL https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS -o $SIGNPATH/KEYS" >> "$LOG_FILE" 2>&1
+  run_check_net_err \
+  "curl --retry 5 --retry-delay 60 -sSL https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS -o $SIGNPATH/KEYS" \
+  "Could not pull signature key file, please ensure connectivity to https://raw.gihubusercontent.com" \
+  noretry >> "$LOG_FILE" 2>&1
   result=$?
   if [[ $result -eq 0 ]]; then
     cat $SIGNPATH/KEYS | gpg --import - >> "$LOG_FILE" 2>&1
-  else
-    echo "Failed to pull signature key file: $result"
-    exit 1
   fi
   
   # Download the containers from the interwebs
@@ -151,14 +149,15 @@ update_docker_containers() {
 
     # Pull down the trusted docker image
     local image=$i:$VERSION$IMAGE_TAG_SUFFIX
-    retry 50 10 "docker pull $CONTAINER_REGISTRY/$IMAGEREPO/$image" >> "$LOG_FILE" 2>&1 
+    run_check_net_err \
+    "docker pull $CONTAINER_REGISTRY/$IMAGEREPO/$image" \
+    "Could not pull $image, please ensure connectivity to $CONTAINER_REGISTRY" >> "$LOG_FILE" 2>&1 
     
     # Get signature
-    retry 50 10 "curl -A '$CURLTYPE/$CURRENTVERSION/$OS/$(uname -r)' https://sigs.securityonion.net/$VERSION/$i:$VERSION$IMAGE_TAG_SUFFIX.sig --output $SIGNPATH/$image.sig" >> "$LOG_FILE" 2>&1
-    if [[ $? -ne 0 ]]; then
-      echo "Unable to pull signature file for $image" >> "$LOG_FILE" 2>&1 
-      exit 1
-    fi
+    run_check_net_err \
+    "curl --retry 5 --retry-delay 60 -A '$CURLTYPE/$CURRENTVERSION/$OS/$(uname -r)' https://sigs.securityonion.net/$VERSION/$i:$VERSION$IMAGE_TAG_SUFFIX.sig --output $SIGNPATH/$image.sig" \
+    "Could not pull signature file for $image, please ensure connectivity to https://sigs.securityonion.net " \
+    noretry >> "$LOG_FILE" 2>&1
     # Dump our hash values
     DOCKERINSPECT=$(docker inspect $CONTAINER_REGISTRY/$IMAGEREPO/$image)
        

salt/common/tools/sbin/so-image-pull

@@ -0,0 +1,58 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+. /usr/sbin/so-image-common
+
+usage() {
+	read -r -d '' message <<- EOM
+		usage: so-image-pull [-h] IMAGE [IMAGE ...]
+
+		positional arguments:
+			IMAGE         One or more 'so-' prefixed images to download and verify.
+
+		optional arguments:
+			-h, --help    Show this help message and exit.
+	EOM
+	echo "$message"
+	exit 1
+}
+
+for arg; do
+	shift
+	[[ "$arg" = "--quiet" || "$arg" = "-q" ]] && quiet=true && continue
+	set -- "$@" "$arg"
+done
+
+if [[ $# -eq 0 || $# -gt 1 ]] || [[ $1 == '-h' || $1 == '--help' ]]; then
+	usage
+fi
+
+TRUSTED_CONTAINERS=("$@")
+set_version
+
+for image in "${TRUSTED_CONTAINERS[@]}"; do
+	if ! docker images | grep "$image" | grep ":5000" | grep -q "$VERSION"; then
+		if [[ $quiet == true ]]; then
+			update_docker_containers "$image" "" "" "/dev/null"
+		else
+			update_docker_containers "$image" "" "" "" 
+		fi
+	else
+		echo "$image:$VERSION image exists."
+	fi
+done

salt/common/tools/sbin/so-import-pcap

@@ -132,6 +132,8 @@ for PCAP in "$@"; do
     PCAP_FIXED=`mktemp /tmp/so-import-pcap-XXXXXXXXXX.pcap`
     echo "- attempting to recover corrupted PCAP file"
     pcapfix "${PCAP}" "${PCAP_FIXED}"
+    # Make fixed file world readable since the Suricata docker container will runas a non-root user
+    chmod a+r "${PCAP_FIXED}"
     PCAP="${PCAP_FIXED}"
     TEMP_PCAPS+=(${PCAP_FIXED})
   fi

salt/common/tools/sbin/so-index-list

@@ -15,8 +15,4 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-{% if grains['role'] in ['so-node','so-heavynode'] %}
-curl -X GET -k -L https://localhost:9200/_cat/indices?v
-{% else %}
-curl -X GET -L localhost:9200/_cat/indices?v
-{% endif %}
+{{ ELASTICCURL }} -X GET -k -L "https://localhost:9200/_cat/indices?v&s=index"

salt/common/tools/sbin/so-influxdb-clean

@@ -0,0 +1,53 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+wdurregex="^[0-9]+w$"
+ddurregex="^[0-9]+d$"
+
+echo -e "\nThis script is used to reduce the size of InfluxDB by removing old data and retaining only the duration specified."
+echo "The duration will need to be specified as an integer followed by the duration unit without a space."
+echo -e "\nFor example, to purge all data but retain the past 12 weeks, specify 12w for the duration."
+echo "The duration units are as follows:"
+echo "  w  - week(s)"
+echo "  d  - day(s)"
+
+while true; do
+  echo ""
+  read -p 'Enter the duration of past data that you would like to retain: ' duration
+  duration=$(echo $duration | tr '[:upper:]' '[:lower:]')
+
+  if [[ "$duration" =~ $wdurregex ]] || [[ "$duration" =~ $ddurregex ]]; then
+    break
+  fi
+
+  echo -e "\nInvalid duration."
+done
+
+echo -e "\nInfluxDB will now be cleaned and leave only the past $duration worth of data."
+read -r -p "Are you sure you want to continue? [y/N] " yorn
+if [[ "$yorn" =~ ^([yY][eE][sS]|[yY])$ ]]; then
+  echo -e "\nCleaning InfluxDb and saving only the past $duration. This may could take several minutes depending on how much data needs to be cleaned."
+    if docker exec -t so-influxdb /bin/bash -c "influx -ssl -unsafeSsl -database telegraf -execute \"DELETE FROM /.*/ WHERE \"time\" >= '2020-01-01T00:00:00.0000000Z' AND \"time\" <= now() - $duration\""; then
+      echo -e "\nInfluxDb clean complete."
+    else
+      echo -e "\nSomething went wrong with cleaning InfluxDB. Please verify that the so-influxdb Docker container is running, and check the log at /opt/so/log/influxdb/influxdb.log for any details."
+    fi
+else
+  echo -e "\nExiting as requested."
+fi

salt/common/tools/sbin/so-influxdb-downsample

@@ -0,0 +1,63 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+{%- set role = grains.id.split('_') | last %}
+{%- if role in ['manager', 'managersearch', 'eval', 'standalone'] %}
+  {%- import_yaml 'influxdb/defaults.yaml' as default_settings %}
+  {%- set influxdb = salt['grains.filter_by'](default_settings, default='influxdb', merge=salt['pillar.get']('influxdb', {})) %}
+
+. /usr/sbin/so-common
+
+echo -e "\nThis script is used to reduce the size of InfluxDB by downsampling old data into the so_long_term retention policy."
+
+echo -e "\nInfluxDB will now be downsampled. This could take a few hours depending on how large the database is and hardware resources available."
+read -r -p "Are you sure you want to continue? [y/N] " yorn
+if [[ "$yorn" =~ ^([yY][eE][sS]|[yY])$ ]]; then
+  echo -e "\nDownsampling InfluxDb started at `date`. This may take several hours depending on how much data needs to be downsampled."
+
+  {% for dest_rp in influxdb.downsample.keys() -%}
+    {% for measurement in influxdb.downsample[dest_rp].get('measurements', []) -%}
+
+  day=0
+  startdate=`date`
+  while docker exec -t so-influxdb /bin/bash -c "influx -ssl -unsafeSsl -database telegraf -execute \"SELECT mean(*) INTO \"so_long_term\".\"{{measurement}}\" FROM \"autogen\".\"{{measurement}}\" WHERE \"time\" >= '2020-07-21T00:00:00.0000000Z' + ${day}d AND \"time\" <= '2020-07-21T00:00:00.0000000Z' + $((day+1))d GROUP BY time(5m),*\""; do
+    # why 2020-07-21? 
+    migrationdate=`date -d "2020-07-21 + ${day} days" +"%y-%m-%d"`
+
+    echo "Downsampling of measurement: {{measurement}} from $migrationdate started at $startdate and completed at `date`."
+
+    newdaytomigrate=$(date -d "$migrationdate + 1 days" +"%s")
+    today=$(date +"%s")
+    if [ $newdaytomigrate -ge $today ]; then
+      break
+    else
+      ((day=day+1))
+      startdate=`date`
+      echo -e "\nDownsampling the next day's worth of data for measurement: {{measurement}}."
+    fi
+  done
+
+    {% endfor -%}
+  {% endfor -%}
+
+  echo -e "\nInfluxDb data downsampling complete."
+
+else
+  echo -e "\nExiting as requested."
+fi
+{%- else %}
+echo -e "\nThis script can only be run on a node running InfluxDB."
+{%- endif %}

salt/common/tools/sbin/so-influxdb-drop-autogen

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+echo -e "\nThis script is used to reduce the size of InfluxDB by dropping the autogen retention policy."
+echo "If you want to retain historical data prior to 2.3.60, then this should only be run after you have downsampled your data using so-influxdb-downsample."
+
+echo -e "\nThe autogen retention policy will now be dropped from InfluxDB."
+read -r -p "Are you sure you want to continue? [y/N] " yorn
+if [[ "$yorn" =~ ^([yY][eE][sS]|[yY])$ ]]; then
+  echo -e "\nDropping autogen retention policy."
+    if docker exec -t so-influxdb influx -format json -ssl -unsafeSsl -execute "drop retention policy autogen on telegraf"; then
+      echo -e "\nAutogen retention policy dropped from InfluxDb."
+    else
+      echo -e "\nSomething went wrong dropping then autogen retention policy from InfluxDB. Please verify that the so-influxdb Docker container is running, and check the log at /opt/so/log/influxdb/influxdb.log for any details."
+    fi
+else
+  echo -e "\nExiting as requested."
+fi

salt/common/tools/sbin/so-kibana-config-export

@@ -23,7 +23,9 @@
 KIBANA_HOST={{ MANAGER }}
 KSO_PORT=5601
 OUTFILE="saved_objects.ndjson"
-curl -s -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST -L $KIBANA_HOST:$KSO_PORT/api/saved_objects/_export -d '{ "type": [ "index-pattern", "config", "visualization", "dashboard", "search" ], "excludeExportDetails": false }' > $OUTFILE
+
+SESSIONCOOKIE=$({{ ELASTICCURL }} -c - -X GET http://$KIBANA_HOST:$KSO_PORT/ | grep sid | awk '{print $7}')
+{{ ELASTICCURL }} -b "sid=$SESSIONCOOKIE" -s -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST -L $KIBANA_HOST:$KSO_PORT/api/saved_objects/_export -d '{ "type": [ "index-pattern", "config", "visualization", "dashboard", "search" ], "excludeExportDetails": false }' > $OUTFILE
 
 # Clean up using PLACEHOLDER
 sed -i "s/$KIBANA_HOST/PLACEHOLDER/g" $OUTFILE

salt/common/tools/sbin/so-kibana-space-defaults

@@ -0,0 +1,13 @@
+. /usr/sbin/so-common
+
+wait_for_web_response "http://localhost:5601/app/kibana" "Elastic" 300 "{{ ELASTICCURL }}"
+## This hackery will be removed if using Elastic Auth ##
+
+# Let's snag a cookie from Kibana
+SESSIONCOOKIE=$({{ ELASTICCURL }} -c - -X GET http://localhost:5601/ | grep sid | awk '{print $7}')
+
+# Disable certain Features from showing up in the Kibana UI
+echo
+echo "Setting up default Space:"
+{{ ELASTICCURL }} -b "sid=$SESSIONCOOKIE" -L -X PUT "localhost:5601/api/spaces/space/default" -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -d' {"id":"default","name":"Default","disabledFeatures":["ml","enterpriseSearch","siem","logs","infrastructure","apm","uptime","monitoring","stackAlerts","actions","fleet"]} ' >> /opt/so/log/kibana/misc.log
+echo

salt/common/tools/sbin/so-learn

@@ -0,0 +1,303 @@
+#!/usr/bin/env python3
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from itertools import chain
+from typing import List
+
+import signal
+import sys
+import os
+import re
+import subprocess
+import argparse
+import textwrap
+import yaml
+import multiprocessing
+import docker
+import pty
+
+minion_pillar_dir = '/opt/so/saltstack/local/pillar/minions'
+so_status_conf = '/opt/so/conf/so-status/so-status.conf'
+proc: subprocess.CompletedProcess = None
+
+# Temp store of modules, will likely be broken out into salt
+def get_learn_modules():
+  return {
+    'logscan': { 'cpu_period': get_cpu_period(fraction=0.25), 'enabled': False, 'description': 'Scan log files against pre-trained models to alert on anomalies.' }
+  }
+
+
+def get_cpu_period(fraction: float):
+  multiplier = 10000
+  
+  num_cores = multiprocessing.cpu_count()
+  if num_cores <= 2:
+    fraction = 1.
+  
+  num_used_cores = int(num_cores * fraction)
+  cpu_period = num_used_cores * multiplier
+  return cpu_period
+
+
+def sigint_handler(*_):
+  print('Exiting gracefully on Ctrl-C')
+  if proc is not None: proc.send_signal(signal.SIGINT)
+  sys.exit(1)
+
+
+def find_minion_pillar() -> str:
+  regex = '^.*_(manager|managersearch|standalone|import|eval)\.sls$'
+  
+  result = []
+  for root, _, files in os.walk(minion_pillar_dir):
+    for f_minion_id in files:
+      if re.search(regex, f_minion_id):
+        result.append(os.path.join(root, f_minion_id))
+  
+  if len(result) == 0:
+    print('Could not find manager-type pillar (eval, standalone, manager, managersearch, import). Are you running this script on the manager?', file=sys.stderr)
+    sys.exit(3)
+  elif len(result) > 1:
+    res_str = ', '.join(f'\"{result}\"')
+    print('(This should not happen, the system is in an error state if you see this message.)\n', file=sys.stderr)
+    print('More than one manager-type pillar exists, minion id\'s listed below:', file=sys.stderr)
+    print(f'  {res_str}', file=sys.stderr)
+    sys.exit(3)
+  else:
+    return result[0]
+
+
+def read_pillar(pillar: str):
+  try:
+    with open(pillar, 'r') as pillar_file:
+      loaded_yaml = yaml.safe_load(pillar_file.read())
+      if loaded_yaml is None:
+        print(f'Could not parse {pillar}', file=sys.stderr)
+        sys.exit(3)
+      return loaded_yaml
+  except:
+    print(f'Could not open {pillar}', file=sys.stderr)
+    sys.exit(3)
+
+
+def write_pillar(pillar: str, content: dict):
+  try:
+    with open(pillar, 'w') as pillar_file:
+      yaml.dump(content, pillar_file, default_flow_style=False)
+  except:
+    print(f'Could not open {pillar}', file=sys.stderr)
+    sys.exit(3)
+
+
+def mod_so_status(action: str, item: str):
+  with open(so_status_conf, 'a+') as conf:
+    conf.seek(0)
+    containers = conf.readlines()
+    
+    if f'so-{item}\n' in containers:
+      if action == 'remove': containers.remove(f'so-{item}\n')
+      if action == 'add': pass
+    else:
+      if action == 'remove': pass
+      if action == 'add': containers.append(f'so-{item}\n')
+      
+    [containers.remove(c_name) for c_name in containers if c_name == '\n'] # remove extra newlines
+    
+    conf.seek(0)
+    conf.truncate(0)
+    conf.writelines(containers)
+
+
+def create_pillar_if_not_exist(pillar:str, content: dict):
+  pillar_dict = content
+
+  if pillar_dict.get('learn', {}).get('modules') is None:
+    pillar_dict['learn'] = {}
+    pillar_dict['learn']['modules'] = get_learn_modules()
+    content.update()
+    write_pillar(pillar, content)
+  
+  return content
+
+
+def salt_call(module: str):
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', f'learn.{module}', 'queue=True']
+
+  print(f'  Applying salt state for {module} module...')
+  proc = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+  return_code = proc.returncode
+  if return_code != 0:
+    print(f'  [ERROR] Failed to apply salt state for {module} module.')
+
+  return return_code
+
+
+def pull_image(module: str):
+  container_basename = f'so-{module}'
+
+  client = docker.from_env()
+  image_list = client.images.list(filters={ 'dangling': False })  
+  tag_list = list(chain.from_iterable(list(map(lambda x: x.attrs.get('RepoTags'), image_list))))
+  basename_match = list(filter(lambda x: f'{container_basename}' in x, tag_list))
+  local_registry_match = list(filter(lambda x: ':5000' in x, basename_match))
+
+  if len(local_registry_match) == 0:
+    print(f'Pulling and verifying missing image for {module} (may take several minutes) ...')
+    pull_command = ['so-image-pull', '--quiet', container_basename]
+  
+    proc = subprocess.run(pull_command, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+    return_code = proc.returncode
+    if return_code != 0:
+      print(f'[ERROR] Failed to pull image so-{module}, skipping state.')
+  else:
+    return_code = 0
+  return return_code
+
+
+def apply(module_list: List):
+  return_code = 0
+  for module in module_list:
+    salt_ret = salt_call(module)
+    # Only update return_code if the command returned a non-zero return
+    if salt_ret != 0:
+      return_code = salt_ret
+    
+  return return_code
+
+
+def check_apply(args: dict):
+  if args.apply:
+    print('Configuration updated. Applying changes:')
+    return apply(args.modules)
+  else:
+    message = 'Configuration updated. Would you like to apply your changes now? (y/N) '
+    answer = input(message)
+    while answer.lower() not in [ 'y', 'n', '' ]:
+      answer = input(message)
+    if answer.lower() in [ 'n', '' ]:
+      return 0
+    else:
+      print('Applying changes:')
+      return apply(args.modules)
+
+
+def enable_disable_modules(args, enable: bool):
+  pillar_modules = args.pillar_dict.get('learn', {}).get('modules')
+  pillar_mod_names = args.pillar_dict.get('learn', {}).get('modules').keys()
+  
+  action_str = 'add' if enable else 'remove'
+
+  if 'all' in args.modules:
+    for module, details in pillar_modules.items():
+      details['enabled'] = enable
+      mod_so_status(action_str, module)
+      if enable: pull_image(module)
+    args.pillar_dict.update()
+    write_pillar(args.pillar, args.pillar_dict)
+  else:
+    write_needed = False
+    for module in args.modules:
+      if module in pillar_mod_names:
+        if pillar_modules[module]['enabled'] == enable:
+          state_str = 'enabled' if enable else 'disabled'
+          print(f'{module} module already {state_str}.', file=sys.stderr)
+        else:
+          if enable and pull_image(module) != 0:
+            continue
+          pillar_modules[module]['enabled'] = enable
+          mod_so_status(action_str, module)
+          write_needed = True
+    if write_needed:
+      args.pillar_dict.update()
+      write_pillar(args.pillar, args.pillar_dict)
+  
+  cmd_ret = check_apply(args)
+  return cmd_ret
+
+
+def enable_modules(args):
+  enable_disable_modules(args, enable=True)
+
+
+def disable_modules(args):
+  enable_disable_modules(args, enable=False)
+
+
+def list_modules(*_):
+  print('Available ML modules:')
+  for module, details in get_learn_modules().items():
+    print(f'  - { module } : {details["description"]}')
+  return 0
+
+
+def main():
+  beta_str = 'BETA - SUBJECT TO CHANGE\n'
+
+  apply_help='After ACTION the chosen modules, apply any necessary salt states.'
+  enable_apply_help = apply_help.replace('ACTION', 'enabling')
+  disable_apply_help = apply_help.replace('ACTION', 'disabling')
+
+  signal.signal(signal.SIGINT, sigint_handler)
+
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
+
+  subcommand_desc = textwrap.dedent(
+    """\
+      enable              Enable one or more ML modules.
+      disable             Disable one or more ML modules.
+      list                List all available ML modules.
+    """
+  )
+
+  subparsers = main_parser.add_subparsers(title='commands', description=subcommand_desc, metavar='', dest='command')
+
+  module_help_str = 'One or more ML modules, which can be listed using \'so-learn list\'. Use the keyword \'all\' to apply the action to all available modules.'
+
+  enable = subparsers.add_parser('enable')
+  enable.set_defaults(func=enable_modules)
+  enable.add_argument('modules', metavar='ML_MODULE', nargs='+', help=module_help_str)
+  enable.add_argument('--apply', action='store_const', const=True, required=False, help=enable_apply_help)
+
+  disable = subparsers.add_parser('disable')
+  disable.set_defaults(func=disable_modules)
+  disable.add_argument('modules', metavar='ML_MODULE', nargs='+', help=module_help_str)
+  disable.add_argument('--apply', action='store_const', const=True, required=False, help=disable_apply_help)
+
+  list = subparsers.add_parser('list')
+  list.set_defaults(func=list_modules)
+
+  args = main_parser.parse_args(sys.argv[1:])
+  args.pillar = find_minion_pillar()
+  args.pillar_dict = create_pillar_if_not_exist(args.pillar, read_pillar(args.pillar))
+
+  if hasattr(args, 'func'):
+    exit_code = args.func(args)
+  else:
+    if args.command is None:
+      print(beta_str)
+      main_parser.print_help()
+    sys.exit(0)
+
+  sys.exit(exit_code)
+
+
+if __name__ == '__main__':
+  main()

salt/common/tools/sbin/so-logstash-events

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        for i in $(curl -s -L http://{{ NODEIP }}:9600/_node/stats | jq .pipelines | jq '. | to_entries | .[].key' | sed 's/\"//g'); do echo ${i^}:; curl -s localhost:9600/_node/stats | jq .pipelines.$i.events; done
+else
+        curl -s -L http://{{ NODEIP }}:9600/_node/stats | jq .pipelines.$1.events
+fi

salt/common/tools/sbin/so-logstash-pipeline-stats

@@ -0,0 +1,25 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+
+. /usr/sbin/so-common
+
+if [ "$1" == "" ]; then
+        curl -s -L http://{{ NODEIP }}:9600/_node/stats | jq .pipelines 
+else
+        curl -s -L http://{{ NODEIP }}:9600/_node/stats | jq .pipelines.$1
+fi

salt/common/tools/sbin/so-pcap-export

@@ -0,0 +1,26 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if [ $# -lt 2 ]; then
+  echo "Usage: $0 <steno-query> Output-Filename"
+  exit 1
+fi
+
+docker exec -it so-sensoroni scripts/stenoquery.sh "$1" -w /nsm/pcapout/$2.pcap
+
+echo ""
+echo "If successful, the output was written to: /nsm/pcapout/$2.pcap"

salt/common/tools/sbin/so-playbook-reset

@@ -22,5 +22,5 @@ salt-call state.apply playbook.db_init,playbook,playbook.automation_user_create
 /usr/sbin/so-soctopus-restart
 
 echo "Importing Plays - this will take some time...."
-wait 5
-/usr/sbin/so-playbook-ruleupdate
+sleep 5
+/usr/sbin/so-playbook-ruleupdate

salt/common/tools/sbin/so-playbook-sync

@@ -17,4 +17,8 @@
 
 . /usr/sbin/so-common
 
+# Check to see if we are already running
+IS_RUNNING=$(ps aux | pgrep -f "so-playbook-sync" | wc -l)
+[ "$IS_RUNNING" -gt 3 ] && echo "$(date) - Multiple Playbook Sync processes already running...exiting." && exit 0
+
 docker exec so-soctopus python3 playbook_play-sync.py

salt/common/tools/sbin/so-raid-status

@@ -17,65 +17,98 @@
 
 . /usr/sbin/so-common
 
-#check_boss_raid() {
-#    BOSSBIN=/opt/boss/mvcli
-#    BOSSRC=$($BOSSBIN info -o vd | grep functional)
-#
-#    if [[ $BOSSRC ]]; then
-#        # Raid is good
-#        BOSSRAID=0
-#    else
-#        BOSSRAID=1
-#    fi
-#}
+appliance_check() {
+  {%- if salt['grains.get']('sosmodel', '') %}
+  APPLIANCE=1
+  DUDEYOUGOTADELL=$(dmidecode |grep Dell)
+  if [[ -n $DUDEYOUGOTADELL ]]; then
+  APPTYPE=dell
+  else
+  APPTYPE=sm
+  fi
+  mkdir -p /opt/so/log/raid
 
-check_lsi_raid() {
-    # For use for LSI on Ubuntu
-    #MEGA=/opt/MegaRAID/MegeCli/MegaCli64
-    #LSIRC=$($MEGA -LDInfo -Lall -aALL | grep Optimal)
-    # Open Source Centos
-    MEGA=/opt/mega/megasasctl
-    LSIRC=$($MEGA | grep optimal)
+  {%- else %}
+  echo "This is not an appliance"
+  exit 0
+  {%- endif %}
+}
 
-    if [[ $LSIRC ]]; then
-        # Raid is good
-        LSIRAID=0
+check_nsm_raid() {
+  PERCCLI=$(/opt/raidtools/perccli/perccli64 /c0/v0 show|grep RAID|grep Optl)
+  MEGACTL=$(/opt/raidtools/megasasctl |grep optimal)
+
+  if [[ $APPLIANCE == '1' ]]; then
+    if [[ -n $PERCCLI ]]; then
+      HWRAID=0
+    elif [[ -n $MEGACTL ]]; then
+      HWRAID=0
     else
-        LSIRAID=1
+      HWRAID=1
     fi
 
+  fi
+
+}
+
+check_boss_raid() {
+  MVCLI=$(/usr/local/bin/mvcli info -o vd |grep status |grep functional)
+
+  if [[ -n $DUDEYOUGOTADELL ]]; then
+    if [[ -n $MVCLI ]]; then
+      BOSSRAID=0
+    else
+      BOSSRAID=1
+    fi
+  fi
 }
 
 check_software_raid() {
+  if [[ -n $DUDEYOUGOTADELL ]]; then
     SWRC=$(grep "_" /proc/mdstat)
 
-    if [[ $SWRC ]]; then
+    if [[ -n $SWRC ]]; then
         # RAID is failed in some way
         SWRAID=1
     else
         SWRAID=0
     fi
+  fi
 }
 
 # This script checks raid status if you use SO appliances
 
 # See if this is an appliance
 
+appliance_check
+check_nsm_raid
+check_boss_raid
 {%- if salt['grains.get']('sosmodel', '') %}
-mkdir -p /opt/so/log/raid
-  {%- if grains['sosmodel'] in ['SOSMN', 'SOSSNNV'] %}
-#check_boss_raid
+{%- if grains['sosmodel'] in ['SOSMN', 'SOSSNNV'] %}
 check_software_raid
-echo "osraid=$BOSSRAID nsmraid=$SWRAID" > /opt/so/log/raid/status.log
-  {%- elif grains['sosmodel'] in ['SOS1000F', 'SOS1000', 'SOSSN7200', 'SOS10K', 'SOS4000'] %}
-#check_boss_raid
-check_lsi_raid
-echo "osraid=$BOSSRAID nsmraid=$LSIRAID" > /opt/so/log/raid/status.log
-  {%- else %}
-exit 0
-  {%- endif %}
-{%- else %}
-exit 0
+{%- endif %}
 {%- endif %}
 
+if [[ -n $SWRAID ]]; then
+  if [[ $SWRAID == '0' && BOSSRAID == '0' ]]; then
+    RAIDSTATUS=0
+  else
+    RAIDSTATUS=1
+  fi
+elif [[ -n $DUDEYOUGOTADELL ]]; then
+  if [[ $BOSSRAID == '0' && $HWRAID == '0' ]]; then
+    RAIDSTATUS=0
+  else
+    RAIDSTATUS=1
+  fi
+elif [[ "$APPTYPE" == 'sm' ]]; then
+  if [[ -n "$HWRAID" ]]; then
+    RAIDSTATUS=0
+  else
+    RAIDSTATUS=1
+  fi
+fi
+
+echo "nsmraid=$RAIDSTATUS" > /opt/so/log/raid/status.log
+
 

salt/common/tools/sbin/so-rule

@@ -37,11 +37,9 @@ def print_err(string: str):
 
 
 def check_apply(args: dict, prompt: bool = True):
-  cmd_arr = ['salt-call', 'state.apply', 'idstools', 'queue=True']
-
   if args.apply:
-    print('Configuration updated. Applying idstools state...')
-    return subprocess.run(cmd_arr)
+    print('Configuration updated. Applying changes:')
+    return apply()
   else:
     if prompt:
       message = 'Configuration updated. Would you like to apply your changes now? (y/N) '
@@ -51,12 +49,24 @@ def check_apply(args: dict, prompt: bool = True):
       if answer.lower() in [ 'n', '' ]:
         return 0
       else:
-        print('Applying idstools state...')
-        return subprocess.run(cmd_arr)
+        print('Applying changes:')
+        return apply()
     else:
       return 0
 
 
+def apply():
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'idstools.sync_files', 'queue=True']
+  update_cmd = ['so-rule-update']
+  print('Syncing config files...')
+  cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  if cmd.returncode == 0:
+    print('Updating rules...')
+    return subprocess.run(update_cmd).returncode
+  else:
+    return cmd.returncode
+
+
 def find_minion_pillar() -> str:
   regex = '^.*_(manager|managersearch|standalone|import|eval)\.sls$'
   
@@ -442,10 +452,7 @@ def main():
         modify.print_help()
     sys.exit(0)
   
-  if isinstance(exit_code, subprocess.CompletedProcess):
-    sys.exit(exit_code.returncode)
-  else:
-    sys.exit(exit_code)
+  sys.exit(exit_code)
 
 
 if __name__ == '__main__':

salt/common/tools/sbin/so-sensor-clean

@@ -115,8 +115,8 @@ clean() {
 }
 
 # Check to see if we are already running
-IS_RUNNING=$(ps aux | grep "so-sensor-clean" | grep -v grep | wc -l)
-[ "$IS_RUNNING" -gt 2 ] && echo "$(date) - $IS_RUNNING sensor clean script processes running...exiting." >>$LOG && exit 0
+IS_RUNNING=$(ps aux | pgrep -f "so-sensor-clean" | wc -l)
+[ "$IS_RUNNING" -gt 3 ] && echo "$(date) - $IS_RUNNING sensor clean script processes running...exiting." >>$LOG && exit 0
 
 if [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; then
 	while [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; do

salt/common/tools/sbin/so-ssh-harden

@@ -4,90 +4,184 @@
 
 if [[ $1 =~ ^(-q|--quiet) ]]; then
 	quiet=true
+elif [[ $1 =~ ^(-v|--verbose) ]]; then
+	verbose=true
 fi
 
+sshd_config=/etc/ssh/sshd_config
+temp_config=/tmp/sshd_config
+
 before=
 after=
 reload_required=false
+change_header_printed=false
 
-print_sshd_t() {
+check_sshd_t() {
 	local string=$1
-	local state=$2
-	echo "${state}:"
 
 	local grep_out
 	grep_out=$(sshd -T | grep "^${string}")
 
-	if [[ $state == "Before" ]]; then
-		before=$grep_out
+	before=$grep_out
+}
+
+print_diff() {
+	local diff
+	diff=$(diff -dbB <(echo $before) <(echo $after) | awk 'NR>1')
+
+	if [[ -n $diff ]]; then
+		if [[ $change_header_printed == false ]]; then
+			printf '%s\n' '' "Changes" '-------' ''
+			change_header_printed=true
+		fi
+		echo -e "$diff\n"
+	fi
+}
+
+replace_or_add() {
+	local type=$1
+	local string=$2
+	if grep -q "$type" $temp_config; then
+		sed -i "/$type .*/d" $temp_config
+	fi
+	printf "%s\n\n" "$string" >> $temp_config
+	reload_required=true
+}
+
+test_config() {
+	local msg
+	msg=$(sshd -t -f $temp_config)
+	local ret=$?
+
+	if [[ -n $msg ]]; then
+		echo "Error found in temp sshd config:"
+		echo $msg
+	fi
+
+	return $ret
+}
+
+main() {
+	if ! [[ $quiet ]]; then echo "Copying current config to $temp_config"; fi
+	cp $sshd_config $temp_config
+
+	# Add newline to ssh for legibility
+	echo "" >> $temp_config
+
+	# Ciphers
+	check_sshd_t "ciphers"
+
+	local bad_ciphers=(
+		"3des-cbc"
+		"aes128-cbc"
+		"aes192-cbc"
+		"aes256-cbc"
+		"arcfour"
+		"arcfour128"
+		"arcfour256"
+		"blowfish-cbc"
+		"cast128-cbc"
+	)
+
+	local cipher_string=$before
+	for cipher in "${bad_ciphers[@]}"; do
+		cipher_string=$(echo "$cipher_string" | sed "s/${cipher}\(,\|\$\)//g" | sed 's/,$//')
+	done
+
+	after=$cipher_string
+
+	if [[ $verbose ]]; then print_diff; fi
+
+	if [[ $before != "$after" ]]; then
+		replace_or_add "ciphers" "$cipher_string" && test_config || exit 1
+	fi
+
+	# KexAlgorithms
+	check_sshd_t "kexalgorithms"
+
+	local bad_kexalgs=(
+		"diffie-hellman-group-exchange-sha1"
+		"diffie-hellman-group-exchange-sha256"
+		"diffie-hellman-group1-sha1"
+		"diffie-hellman-group14-sha1"
+		"ecdh-sha2-nistp256"
+		"ecdh-sha2-nistp521"
+		"ecdh-sha2-nistp384"
+	)
+
+	local kexalg_string=$before
+	for kexalg in "${bad_kexalgs[@]}"; do
+		kexalg_string=$(echo "$kexalg_string" | sed "s/${kexalg}\(,\|\$\)//g" | sed 's/,$//')
+	done
+
+	after=$kexalg_string
+
+	if [[ $verbose ]]; then print_diff; fi
+
+	if [[ $before != "$after" ]]; then
+		replace_or_add "kexalgorithms" "$kexalg_string" && test_config || exit 1
+	fi
+
+	# Macs
+	check_sshd_t "macs"
+	
+	local bad_macs=(
+		"hmac-sha2-512"
+		"umac-128@openssh.com"
+		"hmac-sha2-256"
+		"umac-64@openssh.com"
+		"hmac-sha1"
+		"hmac-sha1-etm@openssh.com"
+		"umac-64-etm@openssh.com"
+	)
+
+	local macs_string=$before
+	for mac in "${bad_macs[@]}"; do
+		macs_string=$(echo "$macs_string" | sed "s/${mac}\(,\|\$\)//g" | sed 's/,$//')
+	done
+
+	after=$macs_string
+
+	if [[ $verbose ]]; then print_diff; fi
+
+	if [[ $before != "$after" ]]; then
+		replace_or_add "macs" "$macs_string" && test_config || exit 1
+	fi
+
+	# HostKeyAlgorithms
+	check_sshd_t "hostkeyalgorithms"
+	
+	local optional_suffix_regex_hka="\(-cert-v01@openssh.com\)\?"
+	local bad_hostkeyalg_list=(
+		"ecdsa-sha2-nistp256"
+		"ecdsa-sha2-nistp384"
+		"ecdsa-sha2-nistp521"
+		"ssh-rsa"
+		"ssh-dss"
+	)
+
+	local hostkeyalg_string=$before
+	for alg in "${bad_hostkeyalg_list[@]}"; do
+		hostkeyalg_string=$(echo "$hostkeyalg_string" | sed "s/${alg}${optional_suffix_regex_hka}\(,\|\$\)//g" | sed 's/,$//')
+	done
+
+	after=$hostkeyalg_string
+	
+	if [[ $verbose ]]; then print_diff; fi
+	
+	if [[ $before != "$after" ]]; then
+		replace_or_add "hostkeyalgorithms" "$hostkeyalg_string" && test_config || exit 1
+	fi
+
+	if [[ $reload_required == true ]]; then
+		mv -f $temp_config $sshd_config
+		if ! [[ $quiet ]]; then echo "Reloading sshd to load config changes"; fi
+		systemctl reload sshd
+		echo "[ WARNING ] Any new ssh sessions will need to remove and reaccept the host key fingerprint for this server before reconnecting."
 	else
-		after=$grep_out
-	fi
-
-	echo $grep_out
-}
-
-print_msg() {
-	local msg=$1
-	if ! [[ $quiet ]]; then
-	printf "%s\n" \
-		"----" \
-		"$msg" \
-		"----" \
-		""
+		if ! [[ $quiet ]]; then echo "No changes made to temp file, cleaning up"; fi
+		rm -f $temp_config
 	fi
 }
 
-if ! [[ $quiet ]]; then print_sshd_t "ciphers" "Before"; fi
-sshd -T | grep "^ciphers" | sed -e "s/\(3des-cbc\|aes128-cbc\|aes192-cbc\|aes256-cbc\|arcfour\|arcfour128\|arcfour256\|blowfish-cbc\|cast128-cbc\|rijndael-cbc@lysator.liu.se\)\,\?//g" >> /etc/ssh/sshd_config
-if ! [[ $quiet ]]; then
-	print_sshd_t "ciphers" "After"
-	echo ""
-fi
-
-if [[ $before != $after ]]; then
-	reload_required=true
-fi
-
-if ! [[ $quiet ]]; then print_sshd_t "kexalgorithms" "Before"; fi
-sshd -T | grep "^kexalgorithms" | sed -e "s/\(diffie-hellman-group14-sha1\|ecdh-sha2-nistp256\|diffie-hellman-group-exchange-sha256\|diffie-hellman-group1-sha1\|diffie-hellman-group-exchange-sha1\|ecdh-sha2-nistp521\|ecdh-sha2-nistp384\)\,\?//g" >> /etc/ssh/sshd_config
-if ! [[ $quiet ]]; then
-	print_sshd_t "kexalgorithms" "After"
-	echo ""
-fi
-
-if [[ $before != $after ]]; then
-	reload_required=true
-fi
-
-if ! [[ $quiet ]]; then print_sshd_t "macs" "Before"; fi
-sshd -T | grep "^macs" | sed -e "s/\(hmac-sha2-512,\|umac-128@openssh.com,\|hmac-sha2-256,\|umac-64@openssh.com,\|hmac-sha1,\|hmac-sha1-etm@openssh.com,\|umac-64-etm@openssh.com,\|hmac-sha1\)//g" >> /etc/ssh/sshd_config
-if ! [[ $quiet ]]; then
-	print_sshd_t "macs" "After"
-	echo ""
-fi
-
-if [[ $before != $after ]]; then
-	reload_required=true
-fi
-
-if ! [[ $quiet ]]; then print_sshd_t "hostkeyalgorithms" "Before"; fi
-sshd -T | grep "^hostkeyalgorithms" | sed "s|ecdsa-sha2-nistp256,||g" | sed "s|ssh-rsa,||g" >> /etc/ssh/sshd_config
-if ! [[ $quiet ]]; then
-	print_sshd_t "hostkeyalgorithms" "After"
-	echo ""
-fi
-
-if [[ $before != $after ]]; then
-	reload_required=true
-fi
-
-if [[ $reload_required == true ]]; then
-	print_msg "Reloading sshd to load config changes..."
-	systemctl reload sshd
-fi
-
-{% if grains['os'] != 'CentOS' %}
-print_msg "[ WARNING ] Any new ssh sessions will need to remove and reaccept the ECDSA key for this server before reconnecting."
-{% endif %}
-
+main

salt/common/tools/sbin/so-suricata-testrule

@@ -23,6 +23,11 @@ TESTPCAP=$2
 
 . /usr/sbin/so-common
 
+if [ $# -lt 2 ]; then
+  echo "Usage: $0 <CustomRule> <TargetPCAP>"
+  exit 1
+fi
+
 echo ""
 echo "==============="
 echo "Running all.rules and $TESTRULE against the following pcap: $TESTPCAP" 

salt/common/tools/sbin/so-tcpreplay

@@ -31,7 +31,7 @@ if [[ $# -lt 1 ]]; then
 	echo "Usage: $0 <pcap-sample(s)>"
 	echo 
 	echo "All PCAPs must be placed in the /opt/so/samples directory unless replaying"
-	echo "a sample pcap that is included in the so-tcpreplay image. Those PCAP sampes"
+	echo "a sample pcap that is included in the so-tcpreplay image. Those PCAP samples"
 	echo "are located in the /opt/samples directory inside of the image."
 	echo 
 	echo "Customer provided PCAP example:"

salt/common/tools/sbin/so-thehive-user-update

@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name>"
+    echo ""
+    echo "Update password for an existing TheHive user. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+THEHIVE_KEY=$(lookup_pillar hivekey)
+THEHVIE_API_URL="$(lookup_pillar url_base)/thehive/api"
+THEHIVE_USER=$USER
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -rs THEHIVE_PASS
+
+if ! check_password "$THEHIVE_PASS"; then
+  echo "Password is invalid. Please exclude single quotes, double quotes and backslashes from the password."
+  exit 2
+fi
+
+# Change password for user in TheHive
+resp=$(curl -sk -XPOST -H "Authorization: Bearer $THEHIVE_KEY" -H "Content-Type: application/json" -L "https://$THEHVIE_API_URL/user/${THEHIVE_USER}/password/set" -d "{\"password\" : \"$THEHIVE_PASS\"}")
+if [[ -z "$resp" ]]; then
+    echo "Successfully updated TheHive user password"
+else
+    echo "Unable to update TheHive user password"
+    echo $resp
+    exit 2
+fi

salt/common/tools/sbin/so-user

@@ -39,10 +39,18 @@ email=$2
 
 kratosUrl=${KRATOS_URL:-http://127.0.0.1:4434}
 databasePath=${KRATOS_DB_PATH:-/opt/so/conf/kratos/db/db.sqlite}
-argon2Iterations=${ARGON2_ITERATIONS:-3}
-argon2Memory=${ARGON2_MEMORY:-14}
-argon2Parallelism=${ARGON2_PARALLELISM:-2}
-argon2HashSize=${ARGON2_HASH_SIZE:-32}
+bcryptRounds=${BCRYPT_ROUNDS:-12}
+elasticUsersFile=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+elasticRolesFile=${ELASTIC_ROLES_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users_roles}
+esUID=${ELASTIC_UID:-930}
+esGID=${ELASTIC_GID:-930}
+
+function lock() {
+  # Obtain file descriptor lock
+  exec 99>/var/tmp/so-user.lock || fail "Unable to create lock descriptor; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  flock -w 10 99 || fail "Another process is using so-user; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  trap 'rm -f /var/tmp/so-user.lock' EXIT
+}
 
 function fail() {
   msg=$1
@@ -58,7 +66,7 @@ function require() {
 
 # Verify this environment is capable of running this script
 function verifyEnvironment() {
-  require "argon2"
+  require "htpasswd"
   require "jq"
   require "curl"
   require "openssl"
@@ -95,6 +103,16 @@ function validateEmail() {
   fi
 }
 
+function hashPassword() {
+  password=$1
+
+  passwordHash=$(echo "${password}" | htpasswd -niBC $bcryptRounds SOUSER)
+  passwordHash=$(echo "$passwordHash" | cut -c 11-)
+  passwordHash="\$2a${passwordHash}" # still waiting for https://github.com/elastic/elasticsearch/issues/51132
+  echo "$passwordHash"
+}
+
+
 function updatePassword() {
   identityId=$1
   
@@ -111,15 +129,125 @@ function updatePassword() {
 
   if [[ -n $identityId ]]; then
     # Generate password hash
-    salt=$(openssl rand -hex 8)
-    passwordHash=$(echo "${password}" | argon2 ${salt} -id -t $argon2Iterations -m $argon2Memory -p $argon2Parallelism -l $argon2HashSize -e)
-
+    passwordHash=$(hashPassword "$password")
     # Update DB with new hash
-    echo "update identity_credentials set config=CAST('{\"hashed_password\":\"${passwordHash}\"}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
+    echo "update identity_credentials set config=CAST('{\"hashed_password\":\"$passwordHash\"}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
     [[ $? != 0 ]] && fail "Unable to update password"
   fi
 }
 
+function createElasticFile() {
+  filename=$1
+  tmpFile=${filename}
+  truncate -s 0 "$tmpFile"
+  chmod 600 "$tmpFile"
+  chown "${esUID}:${esGID}" "$tmpFile"
+}
+
+function syncElasticSystemUser() {
+  json=$1
+  userid=$2
+  usersFile=$3
+
+  user=$(echo "$json" | jq -r ".local.users.$userid.user")
+  pass=$(echo "$json" | jq -r ".local.users.$userid.pass")
+  
+  [[ -z "$user" || -z "$pass" ]] && fail "Elastic auth credentials for system user '$userid' are missing"
+  hash=$(hashPassword "$pass")
+
+  echo "${user}:${hash}" >> "$usersFile"
+}
+
+function syncElasticSystemRole() {
+  json=$1
+  userid=$2
+  role=$3
+  rolesFile=$4
+
+  user=$(echo "$json" | jq -r ".local.users.$userid.user")
+  
+  [[ -z "$user" ]] && fail "Elastic auth credentials for system user '$userid' are missing"
+
+  echo "${role}:${user}" >> "$rolesFile"
+}
+
+function syncElastic() {
+  echo "Syncing users between SOC and Elastic..."
+  usersTmpFile="${elasticUsersFile}.tmp"
+  rolesTmpFile="${elasticRolesFile}.tmp"
+  createElasticFile "${usersTmpFile}"
+  createElasticFile "${rolesTmpFile}"
+
+  authPillarJson=$(lookup_salt_value "auth" "elasticsearch" "pillar" "json")
+
+  syncElasticSystemUser "$authPillarJson" "so_elastic_user"  "$usersTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_elastic_user"  "superuser" "$rolesTmpFile"
+
+  syncElasticSystemUser "$authPillarJson" "so_kibana_user"   "$usersTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_kibana_user"   "superuser" "$rolesTmpFile"
+
+  syncElasticSystemUser "$authPillarJson" "so_logstash_user" "$usersTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_logstash_user" "superuser" "$rolesTmpFile"
+
+  syncElasticSystemUser "$authPillarJson" "so_beats_user"    "$usersTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_beats_user"    "superuser" "$rolesTmpFile"
+
+  syncElasticSystemUser "$authPillarJson" "so_monitor_user"  "$usersTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_monitor_user"  "remote_monitoring_collector" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_monitor_user"  "remote_monitoring_agent" "$rolesTmpFile"
+  syncElasticSystemRole "$authPillarJson" "so_monitor_user"  "monitoring_user" "$rolesTmpFile"
+
+  if [[ -f "$databasePath" ]]; then
+    # Generate the new users file
+    echo "select '{\"user\":\"' || ici.identifier || '\", \"data\":' || ic.config || '}'" \
+      "from identity_credential_identifiers ici, identity_credentials ic " \
+      "where ici.identity_credential_id=ic.id and instr(ic.config, 'hashed_password') " \
+      "order by ici.identifier;" | \
+      sqlite3 "$databasePath" | \
+      jq -r '.user + ":" + .data.hashed_password' \
+      >> "$usersTmpFile"
+    [[ $? != 0 ]] && fail "Unable to read credential hashes from database"
+
+    # Generate the new users_roles file
+    
+    echo "select 'superuser:' || ici.identifier " \
+      "from identity_credential_identifiers ici, identity_credentials ic " \
+      "where ici.identity_credential_id=ic.id and instr(ic.config, 'hashed_password') " \
+      "order by ici.identifier;" | \
+      sqlite3 "$databasePath" \
+      >> "$rolesTmpFile"
+    [[ $? != 0 ]] && fail "Unable to read credential IDs from database"
+  else
+    echo "Database file does not exist yet, skipping users export"
+  fi
+
+  if [[ -s "${usersTmpFile}" ]]; then
+    mv "${usersTmpFile}" "${elasticUsersFile}"
+    mv "${rolesTmpFile}" "${elasticRolesFile}"
+
+    if [[ -z "$SKIP_STATE_APPLY" ]]; then
+      echo "Elastic state will be re-applied to affected minions. This may take several minutes..."
+      echo "Applying elastic state to elastic minions at $(date)" >> /opt/so/log/soc/sync.log 2>&1
+      salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply elasticsearch queue=True >> /opt/so/log/soc/sync.log 2>&1
+    fi
+  else
+    echo "Newly generated users/roles files are incomplete; aborting."
+  fi
+}
+
+function syncAll() {
+  if [[ -z "$FORCE_SYNC" && -f "$databasePath" && -f "$elasticUsersFile" ]]; then
+    usersFileAgeSecs=$(echo $(($(date +%s) - $(date +%s -r "$elasticUsersFile"))))
+    staleCount=$(echo "select count(*) from identity_credentials where updated_at >= Datetime('now', '-${usersFileAgeSecs} seconds');" \
+      | sqlite3 "$databasePath")
+    if [[ "$staleCount" == "0" ]]; then
+      return 1
+    fi
+  fi
+  syncElastic
+  return 0
+}
+
 function listUsers() {
   response=$(curl -Ss -L ${kratosUrl}/identities)
   [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
@@ -178,7 +306,7 @@ function updateStatus() {
     [[ $? != 0 ]] && fail "Unable to unlock credential record"
   fi  
 
-  updatedJson=$(echo "$response" | jq ".traits.status = \"$status\" | del(.verifiable_addresses) | del(.id) | del(.schema_url)")
+  updatedJson=$(echo "$response" | jq ".traits.status = \"$status\" | del(.verifiable_addresses) | del(.id) | del(.schema_url) | del(.created_at) | del(.updated_at)")
   response=$(curl -Ss -XPUT -L ${kratosUrl}/identities/$identityId -d "$updatedJson")
   [[ $? != 0 ]] && fail "Unable to mark user as locked"
 
@@ -208,12 +336,14 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     validateEmail "$email"
     updatePassword
     createUser "$email"
+    syncAll
     echo "Successfully added new user to SOC"
-    check_container thehive && echo $password | so-thehive-user-add "$email"
-    check_container fleet && echo $password | so-fleet-user-add "$email"
+    check_container thehive && echo "$password" | so-thehive-user-add "$email"
+    check_container fleet && echo "$password" | so-fleet-user-add "$email"
     ;;
 
   "list")
@@ -225,7 +355,9 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     updateUser "$email"
+    syncAll
     echo "Successfully updated user"
     ;;
 
@@ -233,7 +365,9 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     updateStatus "$email" 'active'
+    syncAll
     echo "Successfully enabled user"
     check_container thehive && so-thehive-user-enable "$email" true
     check_container fleet && so-fleet-user-enable "$email" true   
@@ -243,7 +377,9 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     updateStatus "$email" 'locked'
+    syncAll
     echo "Successfully disabled user"
     check_container thehive && so-thehive-user-enable "$email" false
     check_container fleet && so-fleet-user-enable "$email" false   
@@ -253,12 +389,19 @@ case "${operation}" in
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
+    lock
     deleteUser "$email"
+    syncAll
     echo "Successfully deleted user"
     check_container thehive && so-thehive-user-enable "$email" false
     check_container fleet && so-fleet-user-enable "$email" false
     ;;
 
+  "sync")
+    lock
+    syncAll
+    ;;
+
   "validate")
     validateEmail "$email"
     updatePassword
@@ -280,4 +423,4 @@ case "${operation}" in
     ;;
 esac
 
-exit 0
+exit 0

salt/common/tools/sbin/so-zeek-logs

@@ -10,11 +10,10 @@ zeek_logs_enabled() {
 }
 
 whiptail_manager_adv_service_zeeklogs() {
-  BLOGS=$(whiptail --title "Security Onion Setup" --checklist "Please Select Logs to Send:" 24 78 12 \
+  BLOGS=$(whiptail --title "so-zeek-logs" --checklist "Please Select Logs to Send:" 24 78 12 \
   "conn" "Connection Logging" ON \
   "dce_rpc" "RPC Logs" ON \
   "dhcp" "DHCP Logs" ON \
-  "dhcpv6" "DHCP IPv6 Logs" ON \
   "dnp3" "DNP3 Logs" ON \
   "dns" "DNS Logs" ON \
   "dpd" "DPD Logs" ON \
@@ -25,25 +24,20 @@ whiptail_manager_adv_service_zeeklogs() {
   "irc" "IRC Chat Logs" ON \
   "kerberos" "Kerberos Logs" ON \
   "modbus" "MODBUS Logs" ON \
-  "mqtt" "MQTT Logs" ON \
   "notice" "Zeek Notice Logs" ON \
   "ntlm" "NTLM Logs" ON \
-  "openvpn" "OPENVPN Logs" ON \
   "pe" "PE Logs" ON \
   "radius" "Radius Logs" ON \
   "rfb" "RFB Logs" ON \
   "rdp" "RDP Logs" ON \
-  "signatures" "Signatures Logs" ON \
   "sip" "SIP Logs" ON \
   "smb_files" "SMB Files Logs" ON \
   "smb_mapping" "SMB Mapping Logs" ON \
   "smtp" "SMTP Logs" ON \
   "snmp" "SNMP Logs" ON \
-  "software" "Software Logs" ON \
   "ssh" "SSH Logs" ON \
   "ssl" "SSL Logs" ON \
   "syslog" "Syslog Logs" ON \
-  "telnet" "Telnet Logs" ON \
   "tunnel" "Tunnel Logs" ON \
   "weird" "Zeek Weird Logs" ON \
   "mysql" "MySQL Logs" ON \
@@ -61,10 +55,10 @@ whiptail_manager_adv_service_zeeklogs
 return_code=$?
 case $return_code in
   1)
-    whiptail --title "Security Onion Setup" --msgbox "Cancelling. No changes have been made." 8 75
+    whiptail --title "so-zeek-logs" --msgbox "Cancelling. No changes have been made." 8 75
     ;;
   255)
-    whiptail --title "Security Onion Setup" --msgbox "Whiptail error occured, exiting." 8 75
+    whiptail --title "so-zeek-logs" --msgbox "Whiptail error occured, exiting." 8 75
     ;;
   *)
     zeek_logs_enabled

salt/common/tools/sbin/so-zeek-stats

@@ -24,11 +24,11 @@ show_stats() {
   echo
   echo "Average throughput:"
   echo
-  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin runuser -l zeek -c '/opt/zeek/bin/zeekctl capstats'
+  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin /opt/zeek/bin/zeekctl capstats
   echo
   echo "Average packet loss:"
   echo
-  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin runuser -l zeek -c '/opt/zeek/bin/zeekctl netstats'
+  docker exec so-zeek env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin:/opt/bin:/usr/local/bin:/usr/local/sbin /opt/zeek/bin/zeekctl netstats
   echo
 }
 

salt/curator/files/bin/so-curator-closed-delete-delete

@@ -4,12 +4,11 @@
 {%- if grains['role'] in ['so-node', 'so-heavynode'] %}
   {%- set ELASTICSEARCH_HOST = salt['pillar.get']('elasticsearch:mainip', '') -%}  
   {%- set ELASTICSEARCH_PORT = salt['pillar.get']('elasticsearch:es_port', '') -%}
-  {%- set LOG_SIZE_LIMIT = salt['pillar.get']('elasticsearch:log_size_limit', '') -%}
-{%- elif grains['role'] in ['so-eval', 'so-managersearch', 'so-standalone'] %}
+{%- elif grains['role'] in ['so-eval', 'so-managersearch', 'so-standalone', 'so-manager'] %}
   {%- set ELASTICSEARCH_HOST = salt['pillar.get']('manager:mainip', '') -%}
   {%- set ELASTICSEARCH_PORT = salt['pillar.get']('manager:es_port', '') -%}
-  {%- set LOG_SIZE_LIMIT = salt['pillar.get']('manager:log_size_limit', '') -%}
 {%- endif -%}
+{%- set LOG_SIZE_LIMIT = salt['pillar.get']('elasticsearch:log_size_limit', '') -%}
 
 # Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
 #
@@ -35,7 +34,7 @@ overlimit() {
 
 closedindices() {
 
-        INDICES=$(curl -s -k {% if grains['role'] in ['so-node','so-heavynode'] %}https://{% endif %}{{ELASTICSEARCH_HOST}}:{{ELASTICSEARCH_PORT}}/_cat/indices?h=index\&expand_wildcards=closed 2> /dev/null)
+        INDICES=$({{ ELASTICCURL }} -s -k https://{{ELASTICSEARCH_HOST}}:{{ELASTICSEARCH_PORT}}/_cat/indices?h=index\&expand_wildcards=closed 2> /dev/null)
         [ $? -eq 1 ] && return false 
         echo ${INDICES} | grep -q -E "(logstash-|so-)"
 }
@@ -50,12 +49,12 @@ while overlimit && closedindices; do
 	# First, get the list of closed indices using _cat/indices?h=index\&expand_wildcards=closed.
 	# Then, sort by date by telling sort to use hyphen as delimiter and then sort on the third field.
 	# Finally, select the first entry in that sorted list.
-	OLDEST_INDEX=$(curl -s -k {% if grains['role'] in ['so-node','so-heavynode'] %}https://{% endif %}{{ELASTICSEARCH_HOST}}:{{ELASTICSEARCH_PORT}}/_cat/indices?h=index\&expand_wildcards=closed | grep -E "(logstash-|so-)" | sort -t- -k3 | head -1)
+	OLDEST_INDEX=$({{ ELASTICCURL }} -s -k https://{{ELASTICSEARCH_HOST}}:{{ELASTICSEARCH_PORT}}/_cat/indices?h=index\&expand_wildcards=closed | grep -E "(logstash-|so-)" | sort -t- -k3 | head -1)
 	
 	# Now that we've determined OLDEST_INDEX, ask Elasticsearch to delete it.
-	curl -XDELETE -k {% if grains['role'] in ['so-node','so-heavynode'] %}https://{% endif %}{{ELASTICSEARCH_HOST}}:{{ELASTICSEARCH_PORT}}/${OLDEST_INDEX}
+	{{ ELASTICCURL }} -XDELETE -k https://{{ELASTICSEARCH_HOST}}:{{ELASTICSEARCH_PORT}}/${OLDEST_INDEX}
 
 	# Finally, write a log entry that says we deleted it.
 	echo "$(date) - Used disk space exceeds LOG_SIZE_LIMIT ({{LOG_SIZE_LIMIT}} GB) - Index ${OLDEST_INDEX} deleted ..." >> ${LOG}
 
-done
+done

salt/curator/files/curator.yml

@@ -1,8 +1,15 @@
 {% if grains['role'] in ['so-node', 'so-heavynode'] %}
   {%- set elasticsearch = salt['pillar.get']('elasticsearch:mainip', '') -%}
-{% elif grains['role'] in ['so-eval', 'so-managersearch', 'so-standalone'] %}
+{% elif grains['role'] in ['so-eval', 'so-managersearch', 'so-standalone', 'so-manager'] %}
   {%- set elasticsearch = salt['pillar.get']('manager:mainip', '') -%}
 {%- endif %}
+{%- if salt['pillar.get']('elasticsearch:auth:enabled') is sameas true %}
+{%-   set ES_USER = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') %}
+{%-   set ES_PASS = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') %}
+{%- else %}
+{%-   set ES_USER = '' %}
+{%-   set ES_PASS = '' %}
+{%- endif %}
 
 ---
 # Remember, leave a key empty if there is no value.  None will be a string,
@@ -11,13 +18,15 @@ client:
   hosts:
     - {{elasticsearch}}
   port: 9200
+{%- if salt['pillar.get']('elasticsearch:auth:enabled') is sameas true %}
+  http_auth: {{ ES_USER }}:{{ ES_PASS }}
+{%- endif %}
   url_prefix:
-{% if grains['role'] in ['so-node', 'so-heavynode'] %}  use_ssl: True{% else %}  use_ssl: False{% endif %}
+  use_ssl: True
   certificate:
   client_cert:
   client_key:
-{% if grains['role'] in ['so-node', 'so-heavynode'] %}  ssl_no_validate: True{% else %}  ssl_no_validate: False{% endif %}
-  http_auth:
+  ssl_no_validate: True
   timeout: 30
   master_only: False
 

salt/curator/init.sls

@@ -4,7 +4,9 @@
 {% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
 {% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
-{% if grains['role'] in ['so-eval', 'so-node', 'so-managersearch', 'so-heavynode', 'so-standalone'] %}
+{% if grains['role'] in ['so-eval', 'so-node', 'so-managersearch', 'so-heavynode', 'so-standalone', 'so-manager'] %}
+  {% from 'elasticsearch/auth.map.jinja' import ELASTICAUTH with context %}
+  {% from "curator/map.jinja" import CURATOROPTIONS with context %}
 # Curator
 # Create the group
 curatorgroup:
@@ -48,6 +50,7 @@ curconf:
     - source: salt://curator/files/curator.yml
     - user: 934
     - group: 939
+    - mode: 660
     - template: jinja
 
 curcloseddel:
@@ -66,6 +69,8 @@ curcloseddeldel:
     - group: 939
     - mode: 755
     - template: jinja
+    - defaults:
+        ELASTICCURL: {{ ELASTICAUTH.elasticcurl }}
 
 curclose:
   file.managed:
@@ -114,8 +119,10 @@ so-curatordeletecron:
    - dayweek: '*'
 
 so-curator:
-  docker_container.running:
+  docker_container.{{ CURATOROPTIONS.status }}:
+  {% if CURATOROPTIONS.status == 'running' %}
     - image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-curator:{{ VERSION }}
+    - start: {{ CURATOROPTIONS.start }}
     - hostname: curator
     - name: so-curator
     - user: curator
@@ -125,11 +132,33 @@ so-curator:
       - /opt/so/conf/curator/curator.yml:/etc/curator/config/curator.yml:ro
       - /opt/so/conf/curator/action/:/etc/curator/action:ro
       - /opt/so/log/curator:/var/log/curator:rw
+    - require:
+      - file: actionconfs
+      - file: curconf
+      - file: curlogdir
+  {% else %}
+    - force: True
+  {% endif %}
 
+  {% if CURATOROPTIONS.manage_sostatus %}
 append_so-curator_so-status.conf:
   file.append:
     - name: /opt/so/conf/so-status/so-status.conf
     - text: so-curator
+    - unless: grep -q so-curator /opt/so/conf/so-status/so-status.conf
+
+    {% if not CURATOROPTIONS.start %}
+so-curator_so-status.disabled:
+  file.comment:
+    - name: /opt/so/conf/so-status/so-status.conf
+    - regex: ^so-curator$
+    {% else %}
+delete_so-curator_so-status.disabled:
+  file.uncomment:
+    - name: /opt/so/conf/so-status/so-status.conf
+    - regex: ^so-curator$
+    {% endif %}
+  {% endif %}
 
 # Begin Curator Cron Jobs
 
@@ -147,4 +176,4 @@ append_so-curator_so-status.conf:
   test.fail_without_changes:
     - name: {{sls}}_state_not_allowed
 
-{% endif %}
+{% endif %}

salt/curator/map.jinja

@@ -0,0 +1,16 @@
+{% set CURATOROPTIONS = {} %}
+{% set ENABLED = salt['pillar.get']('curator:enabled', True) %}
+{% set TRUECLUSTER = salt['pillar.get']('elasticsearch:true_cluster', False) %}
+{% do CURATOROPTIONS.update({'manage_sostatus': True}) %}
+
+# don't start the docker container if searchnode and true clustering is enabled or curator disabled via pillar or true cluster not enabled and manager
+{% if not ENABLED or (TRUECLUSTER and grains.id.split('_')|last == 'searchnode') or (not TRUECLUSTER and grains.id.split('_')|last == 'manager') %}
+  {% do CURATOROPTIONS.update({'start': False}) %}
+  {% do CURATOROPTIONS.update({'status': 'absent'}) %}
+  {% if (TRUECLUSTER and grains.id.split('_')|last == 'searchnode') or (not TRUECLUSTER and grains.id.split('_')|last == 'manager') %}
+    {% do CURATOROPTIONS.update({'manage_sostatus': False}) %}
+  {% endif %}%}
+{% else %}
+  {% do CURATOROPTIONS.update({'start': True}) %}
+  {% do CURATOROPTIONS.update({'status': 'running'}) %}
+{% endif %}

salt/docker_clean/init.sls

@@ -1,86 +1,9 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls in allowed_states %}
 
-{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
-{% set MANAGER = salt['grains.get']('master') %}
-{% set OLDVERSIONS = ['2.0.0-rc.1','2.0.1-rc.1','2.0.2-rc.1','2.0.3-rc.1','2.1.0-rc.2','2.2.0-rc.3','2.3.0','2.3.1','2.3.2','2.3.10','2.3.20']%}
-
-{% for VERSION in OLDVERSIONS %}
-remove_images_{{ VERSION }}:
-  docker_image.absent:
-    - force: True
-    - images:
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-acng:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-thehive-cortex:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-curator:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-domainstats:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elastalert:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elasticsearch:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-filebeat:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-fleet:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-fleet-launcher:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-freqserver:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-grafana:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-idstools:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-influxdb:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-kibana:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-kratos:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-logstash:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-minio:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-mysql:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-nginx:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-playbook:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-redis:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-soc:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-soctopus:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-steno:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-strelka-frontend:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-strelka-manager:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-strelka-backend:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-strelka-filestream:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-suricata:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-telegraf:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-thehive:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-thehive-es:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-wazuh:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/{{ IMAGEREPO }}/so-zeek:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-acng:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-thehive-cortex:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-curator:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-domainstats:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-elastalert:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-elasticsearch:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-filebeat:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-fleet:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-fleet-launcher:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-freqserver:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-grafana:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-idstools:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-influxdb:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-kibana:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-kratos:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-logstash:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-minio:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-mysql:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-nginx:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-pcaptools:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-playbook:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-redis:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-soc:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-soctopus:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-steno:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-strelka-frontend:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-strelka-manager:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-strelka-backend:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-strelka-filestream:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-suricata:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-telegraf:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-thehive:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-thehive-es:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-wazuh:{{ VERSION }}'
-      - '{{ MANAGER }}:5000/securityonion/so-zeek:{{ VERSION }}'
-{% endfor %}
+prune_images:
+  cmd.run:
+    - name: so-docker-prune
 
 {% else %}
 

salt/elastalert/defaults.yaml

@@ -1,3 +1,5 @@
+{%- set ES_USER = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') %}
+{%- set ES_PASS = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') %}
 elastalert:
   config:
     rules_folder: /opt/elastalert/rules/
@@ -16,11 +18,13 @@ elastalert:
     #aws_region: us-east-1
     #profile: test
     #es_url_prefix: elasticsearch
-    #use_ssl: True
-    #verify_certs: True
+    use_ssl: true
+    verify_certs: false
     #es_send_get_body_as: GET
-    #es_username: someusername
-    #es_password: somepassword
+{%- if salt['pillar.get']('elasticsearch:auth:enabled') is sameas true %}
+    es_username: {{ ES_USER }}
+    es_password: {{ ES_PASS }}
+{%- endif %}
     writeback_index: elastalert_status
     alert_time_limit:
       days: 2
@@ -45,4 +49,4 @@ elastalert:
           level: INFO
           handlers:
             - file
-          propagate: false
+          propagate: false

salt/elastalert/files/modules/so/playbook-es.py

@@ -4,21 +4,29 @@ from time import gmtime, strftime
 import requests,json
 from elastalert.alerts import Alerter
 
+import urllib3
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
+
 class PlaybookESAlerter(Alerter):
     """
     Use matched data to create alerts in elasticsearch
     """
 
-    required_options = set(['play_title','play_url','sigma_level','elasticsearch_host'])
+    required_options = set(['play_title','play_url','sigma_level'])
 
     def alert(self, matches):
        for match in matches:
             today = strftime("%Y.%m.%d", gmtime())
-            timestamp = strftime("%Y-%m-%d"'T'"%H:%M:%S", gmtime())
+            timestamp = strftime("%Y-%m-%d"'T'"%H:%M:%S"'.000Z', gmtime())
             headers = {"Content-Type": "application/json"}
+
+            creds = None
+            if 'es_username' in self.rule and 'es_password' in self.rule:
+                creds = (self.rule['es_username'], self.rule['es_password'])
+
             payload = {"rule": { "name": self.rule['play_title'],"case_template": self.rule['play_id'],"uuid": self.rule['play_id'],"category": self.rule['rule.category']},"event":{ "severity": self.rule['event.severity'],"module": self.rule['event.module'],"dataset": self.rule['event.dataset'],"severity_label": self.rule['sigma_level']},"kibana_pivot": self.rule['kibana_pivot'],"soc_pivot": self.rule['soc_pivot'],"play_url": self.rule['play_url'],"sigma_level": self.rule['sigma_level'],"event_data": match, "@timestamp": timestamp}
-            url = f"http://{self.rule['elasticsearch_host']}/so-playbook-alerts-{today}/_doc/"
-            requests.post(url, data=json.dumps(payload), headers=headers, verify=False)
+            url = f"https://{self.rule['es_host']}:{self.rule['es_port']}/so-playbook-alerts-{today}/_doc/"
+            requests.post(url, data=json.dumps(payload), headers=headers, verify=False, auth=creds)
                             
     def get_info(self):
         return {'type': 'PlaybookESAlerter'} 

salt/elastalert/init.sls

@@ -99,13 +99,12 @@ elastaconf:
         elastalert_config: {{ elastalert_config.elastalert.config }}
     - user: 933
     - group: 933
+    - mode: 660
     - template: jinja
 
 wait_for_elasticsearch:
-  module.run:
-    - http.wait_for_successful_query:
-      - url: 'http://{{MANAGER}}:9200/_cat/indices/.kibana*'
-      - wait_for: 180
+  cmd.run:
+    - name: so-elasticsearch-wait
 
 so-elastalert:
   docker_container.running:
@@ -122,7 +121,7 @@ so-elastalert:
     - extra_hosts:
       - {{MANAGER_URL}}:{{MANAGER_IP}}
     - require:
-      - module: wait_for_elasticsearch
+      - cmd: wait_for_elasticsearch
     - watch:
       - file: elastaconf
 

salt/elasticsearch/auth.map.jinja

@@ -0,0 +1,7 @@
+{% set ELASTICAUTH = salt['pillar.filter_by']({
+    True: {
+      'user': salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user'), 
+      'pass': salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass'), 
+      'elasticcurl':'curl -K /opt/so/conf/elasticsearch/curl.config' },
+    False: {'elasticcurl': 'curl'},
+}, pillar='elasticsearch:auth:enabled', default=False) %}

salt/elasticsearch/auth.sls

@@ -0,0 +1,39 @@
+{% set so_elastic_user_pass = salt['random.get_str'](20) %}
+{% set so_kibana_user_pass = salt['random.get_str'](20) %}
+{% set so_logstash_user_pass = salt['random.get_str'](20) %}
+{% set so_beats_user_pass = salt['random.get_str'](20) %}
+{% set so_monitor_user_pass = salt['random.get_str'](20) %}
+
+elastic_auth_pillar:
+  file.managed:
+    - name: /opt/so/saltstack/local/pillar/elasticsearch/auth.sls
+    - mode: 600
+    - reload_pillar: True
+    - contents: |
+        elasticsearch:
+          auth:
+            enabled: False
+            users:
+              so_elastic_user:
+                user: so_elastic
+                pass: {{ so_elastic_user_pass }}
+              so_kibana_user:
+                user: so_kibana
+                pass: {{ so_kibana_user_pass }}
+              so_logstash_user:
+                user: so_logstash
+                pass: {{ so_logstash_user_pass }}
+              so_beats_user:
+                user: so_beats
+                pass: {{ so_beats_user_pass }}
+              so_monitor_user:
+                user: so_monitor
+                pass: {{ so_monitor_user_pass }}
+    # since we are generating a random password, and we don't want that to happen everytime
+    # a highstate runs, we only manage the file each user isn't present in the file. if the
+    # pillar file doesn't exists, then the default vault provided to pillar.get should not
+    # be within the file either, so it should then be created
+    - unless:
+    {% for so_app_user, values in salt['pillar.get']('elasticsearch:auth:users', {'so_noapp_user': {'user': 'r@NDumu53Rd0NtDOoP'}}).items() %}
+      - grep {{ values.user }} /opt/so/saltstack/local/pillar/elasticsearch/auth.sls
+    {% endfor%}

salt/elasticsearch/files/curl.config.template

@@ -0,0 +1 @@
+user = "{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user') }}:{{ salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass') }}"

salt/elasticsearch/files/elasticsearch.yml

@@ -1,6 +1,5 @@
 {%- set NODE_ROUTE_TYPE = salt['pillar.get']('elasticsearch:node_route_type', 'hot') %}
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip') %}
-{%- set FEATURES = salt['pillar.get']('elastic:features', False) %}
 {%- set TRUECLUSTER = salt['pillar.get']('elasticsearch:true_cluster', False) %}
 {%- if TRUECLUSTER is sameas true %}
   {%- set ESCLUSTERNAME = salt['pillar.get']('elasticsearch:true_cluster_name') %}
@@ -10,12 +9,6 @@
 {%- set NODE_ROLES = salt['pillar.get']('elasticsearch:node_roles', ['data', 'ingest']) %}
 cluster.name: "{{ ESCLUSTERNAME }}"
 network.host: 0.0.0.0
-
-# minimum_master_nodes need to be explicitly set when bound on a public IP
-# set to 1 to allow single node clusters
-# Details: https://github.com/elastic/elasticsearch/pull/17288
-#discovery.zen.minimum_master_nodes: 1
-# This is a test -- if this is here, then the volume is mounted correctly.
 path.logs: /var/log/elasticsearch
 action.destructive_requires_name: true
 transport.bind_host: 0.0.0.0
@@ -25,29 +18,27 @@ cluster.routing.allocation.disk.threshold_enabled: true
 cluster.routing.allocation.disk.watermark.low: 95%
 cluster.routing.allocation.disk.watermark.high: 98%
 cluster.routing.allocation.disk.watermark.flood_stage: 98%
-{%- if FEATURES is sameas true %}
 xpack.ml.enabled: false
-#xpack.security.enabled: false
-#xpack.security.transport.ssl.enabled: true
-#xpack.security.transport.ssl.verification_mode: none
-#xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
-#xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
-#xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/ca.crt" ]
-#{%- if grains['role'] in ['so-node','so-heavynode'] %}
-#xpack.security.http.ssl.enabled: true
-#xpack.security.http.ssl.client_authentication: none
-#xpack.security.http.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
-#xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
-#xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/ca.crt
-#{%- endif %}
-#xpack.security.authc:
-#  anonymous:
-#    username: anonymous_user
-#    roles: superuser
-#    authz_exception: true
-{%- endif %}
+xpack.security.enabled: true
+xpack.security.transport.ssl.enabled: true
+xpack.security.transport.ssl.verification_mode: none
+xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
+xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
+xpack.security.transport.ssl.certificate_authorities: [ "/usr/share/elasticsearch/config/ca.crt" ]
+xpack.security.http.ssl.enabled: true
+xpack.security.http.ssl.client_authentication: none
+xpack.security.http.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
+xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
+xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/ca.crt
+{% if not salt['pillar.get']('elasticsearch:auth:enabled', False) %}
+xpack.security.authc:
+  anonymous:
+    username: anonymous_user
+    roles: superuser
+    authz_exception: true
+{% endif %}
 node.name: {{ grains.host }}
-script.max_compilations_rate: 1000/1m
+script.max_compilations_rate: 20000/1m
 {%- if TRUECLUSTER is sameas true %}
   {%- if grains.role == 'so-manager' %}
     {%- if salt['pillar.get']('nodestab', {}) %}
@@ -58,6 +49,16 @@ discovery.seed_hosts:
    - {{ SN.split('_')|first }}
       {%- endfor %}
     {%- endif %}
+  {%- elif grains.role == 'so-managersearch' %}
+      {%- if salt['pillar.get']('nodestab', {}) %}
+node.roles: [ master, data, remote_cluster_client ]
+discovery.seed_hosts:
+   - {{ grains.master }}
+      {%- for SN, SNDATA in salt['pillar.get']('nodestab', {}).items() %}
+   - {{ SN.split('_')|first }}
+      {%- endfor %}
+    {%- endif %}
+node.attr.box_type: {{ NODE_ROUTE_TYPE }}
   {%- else %}
 node.roles: {{ NODE_ROLES }}
 node.attr.box_type: {{ NODE_ROUTE_TYPE }}

salt/elasticsearch/files/ingest/common

@@ -32,8 +32,6 @@
     { "rename":          { "field": "category",            "target_field": "event.category",              "ignore_failure": true, "ignore_missing": true  } },
     { "rename":          { "field": "message2.community_id", "target_field": "network.community_id",  "ignore_failure": true,  "ignore_missing": true  } },
     { "lowercase":       { "field": "event.dataset", "ignore_failure": true,  "ignore_missing": true  } },
-    { "convert":         { "field": "destination.port", "type": "integer",  "ignore_failure": true,  "ignore_missing": true  } },
-    { "convert":         { "field": "source.port", "type": "integer",  "ignore_failure": true,  "ignore_missing": true  } },
     { "convert":         { "field": "log.id.uid", "type": "string",  "ignore_failure": true,  "ignore_missing": true  } },
     { "convert":         { "field": "agent.id", "type": "string",  "ignore_failure": true,  "ignore_missing": true  } },
     { "convert":         { "field": "event.severity", "type": "integer",  "ignore_failure": true,  "ignore_missing": true  } },

salt/elasticsearch/files/ingest/filterlog

@@ -51,7 +51,7 @@
      },
      { "set":         { "field": "_index",        "value": "so-firewall", "override": true           } },
      { "set":         { "if": "ctx.network?.transport_id == '0'", "field": "network.transport",        "value": "icmp", "override": true           } },
-     {"community_id": { "if": "ctx.network?.transport != null", "field":["source.ip","source.port","destination.ip","destination.port","network.transport"],"target_field":"network.community_id"}},
+     {"community_id": {} },
      { "set":         { "field": "module",        "value": "pfsense", "override": true           } },
      { "set":         { "field": "dataset",        "value": "firewall", "override": true           } },
      { "remove":      { "field": ["real_message", "ip_sub_msg", "firewall.sub_message"],     "ignore_failure": true  } }

salt/elasticsearch/files/ingest/http.status

@@ -0,0 +1,70 @@
+{
+  "description" : "http.status",
+  "processors" : [
+	{ "set": { "if": "ctx.http.status_code == 100",    "field": "http.status_message", "value": "Continue" } },
+	{ "set": { "if": "ctx.http.status_code == 101",    "field": "http.status_message", "value": "Switching Protocols" } },
+	{ "set": { "if": "ctx.http.status_code == 102",    "field": "http.status_message", "value": "Processing" } },
+	{ "set": { "if": "ctx.http.status_code == 103",    "field": "http.status_message", "value": "Early Hints" } },
+	{ "set": { "if": "ctx.http.status_code == 200",    "field": "http.status_message", "value": "OK" } },
+	{ "set": { "if": "ctx.http.status_code == 201",    "field": "http.status_message", "value": "Created" } },
+	{ "set": { "if": "ctx.http.status_code == 202",    "field": "http.status_message", "value": "Accepted" } },
+	{ "set": { "if": "ctx.http.status_code == 203",    "field": "http.status_message", "value": "Non-Authoritative Information" } },
+	{ "set": { "if": "ctx.http.status_code == 204",    "field": "http.status_message", "value": "No Content" } },
+	{ "set": { "if": "ctx.http.status_code == 205",    "field": "http.status_message", "value": "Reset Content" } },
+	{ "set": { "if": "ctx.http.status_code == 206",    "field": "http.status_message", "value": "Partial Content" } },
+	{ "set": { "if": "ctx.http.status_code == 207",    "field": "http.status_message", "value": "Multi-Status" } },
+	{ "set": { "if": "ctx.http.status_code == 208",    "field": "http.status_message", "value": "Already Reported" } },
+	{ "set": { "if": "ctx.http.status_code == 226",    "field": "http.status_message", "value": "IM Used" } },
+	{ "set": { "if": "ctx.http.status_code == 300",    "field": "http.status_message", "value": "Multiple Choices" } },
+	{ "set": { "if": "ctx.http.status_code == 301",    "field": "http.status_message", "value": "Moved Permanently" } },
+	{ "set": { "if": "ctx.http.status_code == 302",    "field": "http.status_message", "value": "Found" } },
+	{ "set": { "if": "ctx.http.status_code == 303",    "field": "http.status_message", "value": "See Other" } },
+	{ "set": { "if": "ctx.http.status_code == 304",    "field": "http.status_message", "value": "Not Modified" } },
+	{ "set": { "if": "ctx.http.status_code == 305",    "field": "http.status_message", "value": "Use Proxy" } },
+	{ "set": { "if": "ctx.http.status_code == 306",    "field": "http.status_message", "value": "(Unused)" } },
+	{ "set": { "if": "ctx.http.status_code == 307",    "field": "http.status_message", "value": "Temporary Redirect" } },
+	{ "set": { "if": "ctx.http.status_code == 308",    "field": "http.status_message", "value": "Permanent Redirect" } },
+	{ "set": { "if": "ctx.http.status_code == 400",    "field": "http.status_message", "value": "Bad Request" } },
+	{ "set": { "if": "ctx.http.status_code == 401",    "field": "http.status_message", "value": "Unauthorized" } },
+	{ "set": { "if": "ctx.http.status_code == 402",    "field": "http.status_message", "value": "Payment Required" } },
+	{ "set": { "if": "ctx.http.status_code == 403",    "field": "http.status_message", "value": "Forbidden" } },
+	{ "set": { "if": "ctx.http.status_code == 404",    "field": "http.status_message", "value": "Not Found" } },
+	{ "set": { "if": "ctx.http.status_code == 405",    "field": "http.status_message", "value": "Method Not Allowed" } },
+	{ "set": { "if": "ctx.http.status_code == 406",    "field": "http.status_message", "value": "Not Acceptable" } },
+	{ "set": { "if": "ctx.http.status_code == 407",    "field": "http.status_message", "value": "Proxy Authentication Required" } },
+	{ "set": { "if": "ctx.http.status_code == 408",    "field": "http.status_message", "value": "Request Timeout" } },
+	{ "set": { "if": "ctx.http.status_code == 409",    "field": "http.status_message", "value": "Conflict" } },
+	{ "set": { "if": "ctx.http.status_code == 410",    "field": "http.status_message", "value": "Gone" } },
+	{ "set": { "if": "ctx.http.status_code == 411",    "field": "http.status_message", "value": "Length Required" } },
+	{ "set": { "if": "ctx.http.status_code == 412",    "field": "http.status_message", "value": "Precondition Failed" } },
+	{ "set": { "if": "ctx.http.status_code == 413",    "field": "http.status_message", "value": "Payload Too Large" } },
+	{ "set": { "if": "ctx.http.status_code == 414",    "field": "http.status_message", "value": "URI Too Long" } },
+	{ "set": { "if": "ctx.http.status_code == 415",    "field": "http.status_message", "value": "Unsupported Media Type" } },
+	{ "set": { "if": "ctx.http.status_code == 416",    "field": "http.status_message", "value": "Range Not Satisfiable" } },
+	{ "set": { "if": "ctx.http.status_code == 417",    "field": "http.status_message", "value": "Expectation Failed" } },
+	{ "set": { "if": "ctx.http.status_code == 421",    "field": "http.status_message", "value": "Misdirected Request" } },
+	{ "set": { "if": "ctx.http.status_code == 422",    "field": "http.status_message", "value": "Unprocessable Entity" } },
+	{ "set": { "if": "ctx.http.status_code == 423",    "field": "http.status_message", "value": "Locked" } },
+	{ "set": { "if": "ctx.http.status_code == 424",    "field": "http.status_message", "value": "Failed Dependency" } },
+	{ "set": { "if": "ctx.http.status_code == 425",    "field": "http.status_message", "value": "Too Early" } },
+	{ "set": { "if": "ctx.http.status_code == 426",    "field": "http.status_message", "value": "Upgrade Required" } },
+	{ "set": { "if": "ctx.http.status_code == 427",    "field": "http.status_message", "value": "Unassigned" } },
+	{ "set": { "if": "ctx.http.status_code == 428",    "field": "http.status_message", "value": "Precondition Required" } },
+	{ "set": { "if": "ctx.http.status_code == 429",    "field": "http.status_message", "value": "Too Many Requests" } },
+	{ "set": { "if": "ctx.http.status_code == 430",    "field": "http.status_message", "value": "Unassigned" } },
+	{ "set": { "if": "ctx.http.status_code == 431",    "field": "http.status_message", "value": "Request Header Fields Too Large" } },
+	{ "set": { "if": "ctx.http.status_code == 451",    "field": "http.status_message", "value": "Unavailable For Legal Reasons" } },
+	{ "set": { "if": "ctx.http.status_code == 500",    "field": "http.status_message", "value": "Internal Server Error" } },
+	{ "set": { "if": "ctx.http.status_code == 501",    "field": "http.status_message", "value": "Not Implemented" } },
+	{ "set": { "if": "ctx.http.status_code == 502",    "field": "http.status_message", "value": "Bad Gateway" } },
+	{ "set": { "if": "ctx.http.status_code == 503",    "field": "http.status_message", "value": "Service Unavailable" } },
+	{ "set": { "if": "ctx.http.status_code == 504",    "field": "http.status_message", "value": "Gateway Timeout" } },
+	{ "set": { "if": "ctx.http.status_code == 505",    "field": "http.status_message", "value": "HTTP Version Not Supported" } },
+	{ "set": { "if": "ctx.http.status_code == 506",    "field": "http.status_message", "value": "Variant Also Negotiates" } },
+	{ "set": { "if": "ctx.http.status_code == 507",    "field": "http.status_message", "value": "Insufficient Storage" } },
+	{ "set": { "if": "ctx.http.status_code == 508",    "field": "http.status_message", "value": "Loop Detected" } },
+	{ "set": { "if": "ctx.http.status_code == 509",    "field": "http.status_message", "value": "Unassigned" } },
+	{ "set": { "if": "ctx.http.status_code == 510",    "field": "http.status_message", "value": "Not Extended" } },
+	{ "set": { "if": "ctx.http.status_code == 511",    "field": "http.status_message", "value": "Network Authentication Required" } }
+  ]
+}

salt/elasticsearch/files/ingest/logscan.alert

@@ -0,0 +1,29 @@
+{
+    "description": "logscan",
+    "processors": [
+        { "set": { "field": "event.severity", "value": 2 } },
+        { "json": { "field": "message", "add_to_root": true, "ignore_failure": true } },
+        { "rename": { "field": "@timestamp", "target_field": "event.ingested", "ignore_missing": true } },
+        { "date": { "field": "timestamp", "target_field": "event.created", "formats": [ "ISO8601", "UNIX" ], "ignore_failure": true } },
+        { "date": { "field": "start_time", "target_field": "@timestamp", "formats": [ "ISO8601", "UNIX" ], "ignore_failure": true } },
+        { "date": { "field": "start_time", "target_field": "event.start", "formats": [ "ISO8601", "UNIX" ], "ignore_failure": true } },
+        { "date": { "field": "end_time", "target_field": "event.end", "formats": [ "ISO8601", "UNIX" ], "ignore_failure": true } },
+        { "remove": { "field": "start_time", "ignore_missing": true } },
+        { "remove": { "field": "end_time", "ignore_missing": true } },
+        { "rename": { "field": "source_ip", "target_field": "source.ip", "ignore_missing": true } },
+        { "rename": { "field": "top_source_ips", "target_field": "logscan.source.ips", "ignore_missing": true } },
+        { "append": { "if": "ctx.source != null", "field": "logscan.source.ips", "value": "{{{source.ip}}}", "ignore_failure": true } },
+        { "set": { "if": "ctx.model == 'k1'", "field": "rule.name", "value": "LOGSCAN K1 MODEL THRESHOLD" } },
+        { "set": { "if": "ctx.model == 'k1'", "field": "rule.description", "value": "High number of logins from single IP in 1 minute window" } },
+        { "set": { "if": "ctx.model == 'k5'", "field": "rule.name", "value": "LOGSCAN K5 MODEL THRESHOLD" } },
+        { "set": { "if": "ctx.model == 'k5'", "field": "rule.description", "value": "High ratio of login failures from single IP in 5 minute window" } },
+        { "set": { "if": "ctx.model == 'k60'", "field": "rule.name", "value": "LOGSCAN K60 MODEL THRESHOLD" } },
+        { "set": { "if": "ctx.model == 'k60'", "field": "rule.description", "value": "Large number of login failures in 1 hour window" } },
+        { "rename": { "field": "model", "target_field": "logscan.model" } },
+        { "rename": { "field": "num_attempts", "target_field": "logscan.attempts.total.amount", "ignore_missing": true } },
+        { "rename": { "field": "num_failed", "target_field": "logscan.attempts.failed.amount", "ignore_missing": true } },
+        { "script": { "lang": "painless", "source": "ctx.logscan.attempts.succeeded.amount = ctx.logscan.attempts.total.amount - ctx.logscan.attempts.failed.amount" , "ignore_failure": true} },
+        { "rename": { "field": "avg_failure_interval", "target_field": "logscan.attempts.failed.avg_interval", "ignore_missing": true } },
+        { "pipeline": { "name": "common" } }
+    ]
+}

salt/elasticsearch/files/ingest/osquery.live_query

@@ -0,0 +1,16 @@
+{
+  "description" : "osquery live query",
+  "processors" : [
+    {
+      "script": {
+        "lang": "painless",
+        "source": "def dict = ['columns': new HashMap()]; for (entry in ctx['rows'].entrySet()) { dict['columns'][entry.getKey()] = entry.getValue(); } ctx['result'] = dict; "
+      }
+    },
+    { "remove":      { "field": [ "rows" ], "ignore_missing": true, "ignore_failure": true } },
+    { "rename":      { "field": "distributed_query_execution_id",  "target_field": "result.query_id",       "ignore_missing": true  } },   
+    { "rename":      { "field": "computer_name",  "target_field": "host.hostname",       "ignore_missing": true  } },        
+    { "pipeline":    { "name": "osquery.normalize" } },
+    { "pipeline":    { "name": "common" } }
+  ]
+}

salt/elasticsearch/files/ingest/osquery.normalize

@@ -0,0 +1,14 @@
+{
+  "description" : "osquery normalize",
+  "processors" : [
+     { "rename":      { "field": "result.columns.cmdline",               "target_field": "process.command_line",       "ignore_missing": true  } },    
+     { "rename":      { "field": "result.columns.cwd",               "target_field": "process.working_directory",       "ignore_missing": true  } },   
+     { "rename":      { "field": "result.columns.name",               "target_field": "process.name",       "ignore_missing": true  } },    
+     { "rename":      { "field": "result.columns.path",               "target_field": "process.executable",       "ignore_missing": true  } }, 
+     { "rename":      { "field": "result.columns.pid",               "target_field": "process.pid",       "ignore_missing": true  } },    
+     { "rename":      { "field": "result.columns.parent",               "target_field": "process.ppid",       "ignore_missing": true  } }, 
+     { "rename":      { "field": "result.columns.uid",               "target_field": "user.id",       "ignore_missing": true  } },   
+     { "rename":      { "field": "result.columns.username",           "target_field": "user.name",       "ignore_missing": true  } },    
+     { "rename":      { "field": "result.columns.gid",               "target_field": "group.id",       "ignore_missing": true  } }     
+  ]
+}

salt/elasticsearch/files/ingest/osquery.query_result

@@ -1,24 +1,20 @@
 {
   "description" : "osquery",
   "processors" : [
-    { "json":        { "field": "message",                                            "target_field": "message2",             "ignore_failure": true  } },
-    { "gsub":        { "field": "message2.columns.data",                              "pattern": "\\\\xC2\\\\xAE",             "replacement": "", "ignore_missing": true  } },
-    { "rename":      { "if": "ctx.message2.columns?.eventid != null", "field": "message2.columns",  "target_field": "winlog",       "ignore_missing": true  } },
+    { "json":        { "field": "message",                                            "target_field": "result",             "ignore_failure": true  } },
+    { "gsub":        { "field": "result.columns.data",                              "pattern": "\\\\xC2\\\\xAE",             "replacement": "", "ignore_missing": true  } },
+    { "rename":      { "if": "ctx.result.columns?.eventid != null", "field": "result.columns",  "target_field": "winlog",       "ignore_missing": true  } },
     { "json":        { "field": "winlog.data",    "target_field": "unparsed",      "ignore_failure": true} },
     { "set":         { "if": "!(ctx.unparsed?.EventData instanceof Map)", "field": "error.eventdata_parsing", "value": true, "ignore_failure": true  } },     
     { "rename":      { "if": "!(ctx.error?.eventdata_parsing == true)", "field": "unparsed.EventData", "target_field": "winlog.event_data",       "ignore_missing": true, "ignore_failure": true  } },   
     { "rename":      { "field": "winlog.source",               "target_field": "winlog.channel",       "ignore_missing": true  } },    
     { "rename":      { "field": "winlog.eventid",               "target_field": "winlog.event_id",       "ignore_missing": true  } },   
+    { "rename":      { "field": "winlog.datetime",      "target_field": "winlog.systemTime",     "ignore_missing": true } },
     { "pipeline":    { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational'",   "name": "sysmon"  }  },
-    { "pipeline":    { "if": "ctx.winlog?.channel != 'Microsoft-Windows-Sysmon/Operational'",  "name":"win.eventlogs" }  },
-    {
-      "script": {
-        "lang": "painless",
-        "source": "def dict = ['result': new HashMap()]; for (entry in ctx['message2'].entrySet()) { dict['result'][entry.getKey()] = entry.getValue(); } ctx['osquery'] = dict; "
-      }
-    },
+    { "pipeline":    { "if": "ctx.winlog?.channel != 'Microsoft-Windows-Sysmon/Operational' && ctx.containsKey('winlog')",  "name":"win.eventlogs" }  },
     { "set":           { "field": "event.module", "value": "osquery", "override": false }  },
-    { "set":           { "field": "event.dataset", "value": "{{osquery.result.name}}", "override": false}  },
+    { "set":           { "field": "event.dataset", "value": "{{result.name}}", "override": false}  },
+    { "pipeline":    { "if": "!(ctx.containsKey('winlog'))",  "name": "osquery.normalize" } },
     { "pipeline":    { "name": "common" } }
   ]
 }

salt/elasticsearch/files/ingest/ossec

@@ -33,6 +33,7 @@
     { "rename":         { "field": "data.win.eventdata.user",                   "target_field": "user.name",            "ignore_missing": true  } },
     { "rename":         { "field": "data.win.system",                   "target_field": "winlog",           "ignore_missing": true  } },
     { "rename":         { "field": "data.win.eventdata",                   "target_field": "winlog.event_data",           "ignore_missing": true  } },
+    { "rename":         { "field": "data",                   "target_field": "wazuh.data",           "ignore_missing": true  } },
     { "rename":         { "field": "winlog.eventID",                   "target_field": "winlog.event_id",           "ignore_missing": true  } },
     { "rename":         { "field": "predecoder.program_name",                   "target_field": "process.name",         "ignore_missing": true  } },
     { "rename":         { "field": "decoder.name",                   "target_field": "event.dataset",         "ignore_missing": true  } },

salt/elasticsearch/files/ingest/strelka.file

@@ -8,6 +8,7 @@
     { "rename":         { "field": "scan.hash",        "target_field": "hash",          "ignore_missing": true  } },
     { "rename":         { "field": "scan.exiftool",        "target_field": "exiftool",          "ignore_missing": true  } },
     { "grok":           { "if": "ctx.request?.attributes?.filename != null",   "field": "request.attributes.filename",       "patterns": ["-%{WORD:log.id.fuid}-"], "ignore_failure": true } },
+    { "gsub":           { "if": "ctx.request?.attributes?.filename != null",   "field": "request.attributes.filename",       "pattern": "\/nsm\/strelka\/staging",      "replacement": "\/nsm\/strelka\/processed" } },
     { "foreach":
       {
         "if": "ctx.exiftool?.keys !=null",

salt/elasticsearch/files/ingest/suricata.dhcp

@@ -1,13 +1,14 @@
 {
   "description" : "suricata.dhcp",
   "processors" : [
-    { "rename": 	{ "field": "message2.proto", 		"target_field": "network.transport",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.app_proto", 		"target_field": "network.protocol",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dhcp.assigned_ip", 		"target_field": "dhcp.assigned_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dhcp.client_mac", 		"target_field": "host.mac",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dhcp.dhcp_type", 		"target_field": "dhcp.message_types",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dhcp.assigned_ip", 		"target_field": "dhcp.assigned_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dhcp.type", 		"target_field": "dhcp.type",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.proto", 		"target_field": "network.transport",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.app_proto", 	"target_field": "network.protocol",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dhcp.assigned_ip",	"target_field": "dhcp.assigned_ip",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dhcp.client_ip", 	"target_field": "client.address",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dhcp.client_mac", 	"target_field": "host.mac",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dhcp.dhcp_type", 	"target_field": "dhcp.message_types",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dhcp.hostname", 	"target_field": "host.hostname",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dhcp.type", 	"target_field": "dhcp.type",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.dhcp.id", 		"target_field": "dhcp.id",		"ignore_missing": true 	} },
     { "pipeline": { "name": "common" } }
   ]

salt/elasticsearch/files/ingest/suricata.dns

@@ -12,7 +12,7 @@
     { "rename": 	{ "field": "message2.dns.qr", 		"target_field": "dns.qr",			"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.dns.rd", 		"target_field": "dns.recursion.desired",	"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.dns.ra", 		"target_field": "dns.recursion.available",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dns.rcode", 	"target_field": "dns.response.code",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.dns.rcode", 	"target_field": "dns.response.code_name",	"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.grouped.A", 	"target_field": "dns.answers.data",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.grouped.CNAME", 	"target_field": "dns.answers.name",		"ignore_missing": true 	} },
     { "pipeline": 	{ "if": "ctx.dns.query?.name != null && ctx.dns.query.name.contains('.')", "name": "dns.tld"			} },

salt/elasticsearch/files/ingest/suricata.fileinfo

@@ -13,6 +13,7 @@
     { "rename": 	{ "field": "message2.fileinfo.size", 		"target_field": "file.size",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.fileinfo.state", 		"target_field": "file.state",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.fileinfo.stored", 		"target_field": "file.saved",		"ignore_missing": true 	} },
+    { "rename":   { "field": "message2.fileinfo.sha256",   "target_field": "hash.sha256", "ignore_missing": true } },
     { "set":		{ "if": "ctx.network?.protocol != null", 	"field": "file.source",	"value": "{{network.protocol}}"  	} },
     { "pipeline": { "name": "common" } }
   ]