Merge pull request #7106 from Security-Onion-Solutions/hotfix/2.3.100

Hotfix/2.3.100

CONTRIBUTING.md

@@ -15,7 +15,7 @@
 
 ### Contributing code
 
-* **All commits must be signed** with a valid key that has been added to your GitHub account. The commits should have all the "**Verified**" tag when viewed on GitHub as shown below:
+* **All commits must be signed** with a valid key that has been added to your GitHub account. Each commit should have the "**Verified**" tag when viewed on GitHub as shown below:
   
   <img src="./assets/images/verified-commit-1.png" width="450">
 
@@ -29,6 +29,7 @@
 
 * See this document's [code styling and conventions section](#code-style-and-conventions) below to be sure your PR fits our code requirements prior to submitting.
 
+* Minor bug fixes can be submitted immediately. However, if you are wanting to make more involved changes, please start a [discussion](https://github.com/Security-Onion-Solutions/securityonion/discussions) first and tell us what you are hoping to achieve. If we agree with your goals, then you can submit the PR.
 
 
 ### Code style and conventions

HOTFIX

@@ -1 +1 @@
-
+20220202 20220203

README.md

@@ -1,6 +1,6 @@
-## Security Onion 2.3.61
+## Security Onion 2.3.100
 
-Security Onion 2.3.61 is here!
+Security Onion 2.3.100 is here!
 
 ## Screenshots
 

VERIFY_ISO.md

@@ -1,18 +1,18 @@
-### 2.3.61 ISO image built on 2021/07/22
+### 2.3.100-20220203 ISO image built on 2022/02/03
 
 
 
 ### Download and Verify
 
-2.3.61 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-2.3.61.iso
+2.3.100-20220203 ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-2.3.100-20220203.iso
 
-MD5: 538F29F3AB57087FC879108FFC81447C  
-SHA1: C2239206572CBEB697CFA2A4850A16A54BF5FB0D  
-SHA256: F5035361B63D1EE8D87CE7B0D8333E521A44453274785B62630CAC76C1BEA929 
+MD5: 14705B2F2F9C973D944A4545449799C5  
+SHA1: D73405BE3DE404DE19979B58DEA6F22F434E622D  
+SHA256: 3DD54ACBFDE0047A5EA238415F025ADB6D6AAFF53BEE084A602327CB3242B580 
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.61.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.100-20220203.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS  
@@ -26,22 +26,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/ma
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.61.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.3.100-20220203.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-2.3.61.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-2.3.100-20220203.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-2.3.61.iso.sig securityonion-2.3.61.iso
+gpg --verify securityonion-2.3.100-20220203.iso.sig securityonion-2.3.100-20220203.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Thu 22 Jul 2021 10:28:58 AM EDT using RSA key ID FE507013
+gpg: Signature made Thu 03 Feb 2022 03:35:03 PM EST using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.

VERSION

@@ -1 +1 @@
-2.3.61
+2.3.100

files/firewall/assigned_hostgroups.local.map.yaml

@@ -16,6 +16,7 @@ role:
   import:
   manager:
   managersearch:
+  receiver:
   standalone:
   searchnode:
-  sensor:
+  sensor:

files/firewall/hostgroups.local.yaml

@@ -16,6 +16,10 @@ firewall:
       ips:
         delete:
         insert:
+    endgame:
+      ips:
+        delete:
+        insert:
     fleet:
       ips:
         delete:
@@ -40,6 +44,10 @@ firewall:
       ips:
         delete:
         insert:
+    receiver:
+      ips:
+        delete:
+        insert:
     search_node:
       ips:
         delete:

pillar/elasticsearch/eval.sls

@@ -1,7 +1,8 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-case-template.json.jinja
+    - so/so-common-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja
@@ -10,4 +11,4 @@ elasticsearch:
     - so/so-ossec-template.json.jinja
     - so/so-strelka-template.json.jinja
     - so/so-syslog-template.json.jinja
-    - so/so-zeek-template.json.jinja
+    - so/so-zeek-template.json.jinja

pillar/elasticsearch/manager.sls

@@ -1,7 +1,9 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-case-template.json.jinja
+    - so/so-common-template.json.jinja
+    - so/so-endgame-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/elasticsearch/search.sls

@@ -1,7 +1,9 @@
 elasticsearch:
   templates:
     - so/so-beats-template.json.jinja
-    - so/so-common-template.json
+    - so/so-case-template.json.jinja
+    - so/so-common-template.json.jinja
+    - so/so-endgame-template.json.jinja
     - so/so-firewall-template.json.jinja
     - so/so-flow-template.json.jinja
     - so/so-ids-template.json.jinja

pillar/logstash/init.sls

@@ -1,6 +1,7 @@
 logstash:
   docker_options:
     port_bindings:
+      - 0.0.0.0:3765:3765
       - 0.0.0.0:5044:5044
       - 0.0.0.0:5644:5644
       - 0.0.0.0:6050:6050

pillar/logstash/manager.sls

@@ -1,9 +1,9 @@
-{%- set PIPELINE = salt['pillar.get']('global:pipeline', 'redis') %}
 logstash:
   pipelines:
     manager:
       config:
         - so/0009_input_beats.conf      
         - so/0010_input_hhbeats.conf
+        - so/0011_input_endgame.conf
         - so/9999_output_redis.conf.jinja
         

pillar/logstash/nodes.sls

@@ -0,0 +1,31 @@
+{% set node_types = {} %}
+{% set cached_grains = salt.saltutil.runner('cache.grains', tgt='*') %}
+{% for minionid, ip in salt.saltutil.runner(
+    'mine.get',
+    tgt='G@role:so-manager or G@role:so-managersearch or G@role:so-standalone or G@role:so-node or G@role:so-heavynode or G@role:so-receiver or G@role:so-helix ',
+    fun='network.ip_addrs',
+    tgt_type='compound') | dictsort()
+%}
+
+{%   set hostname = cached_grains[minionid]['host'] %}
+{%   set node_type = minionid.split('_')[1] %}
+{%   if node_type not in node_types.keys() %}
+{%     do node_types.update({node_type: {hostname: ip[0]}}) %}
+{%   else %}
+{%     if hostname not in node_types[node_type] %}
+{%       do node_types[node_type].update({hostname: ip[0]}) %}
+{%     else %}
+{%       do node_types[node_type][hostname].update(ip[0]) %}
+{%     endif %}
+{%   endif %}
+{% endfor %}
+
+logstash:
+  nodes:
+{% for node_type, values in node_types.items() %}
+    {{node_type}}:
+{%   for hostname, ip in values.items() %}
+      {{hostname}}:
+        ip: {{ip}}
+{%   endfor %}
+{% endfor %}

pillar/logstash/receiver.sls

@@ -0,0 +1,9 @@
+logstash:
+  pipelines:
+    receiver:
+      config:
+        - so/0009_input_beats.conf      
+        - so/0010_input_hhbeats.conf
+        - so/0011_input_endgame.conf
+        - so/9999_output_redis.conf.jinja
+        

pillar/logstash/search.sls

@@ -1,4 +1,3 @@
-{%- set PIPELINE = salt['pillar.get']('global:pipeline', 'minio') %}
 logstash:
   pipelines:
     search:
@@ -13,3 +12,5 @@ logstash:
         - so/9500_output_beats.conf.jinja
         - so/9600_output_ossec.conf.jinja
         - so/9700_output_strelka.conf.jinja
+        - so/9800_output_logscan.conf.jinja
+        - so/9900_output_endgame.conf.jinja

pillar/node_data/ips.sls

@@ -0,0 +1,33 @@
+{% set node_types = {} %}
+{% set manage_alived = salt.saltutil.runner('manage.alived', show_ip=True) %}
+{% set manager = grains.master %}
+{% set manager_type = manager.split('_')|last %}
+{% for minionid, ip in salt.saltutil.runner('mine.get', tgt='*', fun='network.ip_addrs', tgt_type='glob') | dictsort() %}
+{%   set hostname = minionid.split('_')[0] %}
+{%   set node_type = minionid.split('_')[1] %}
+{%   set is_alive = False %}
+{%   if minionid in manage_alived.keys() %}
+{%     if ip[0] == manage_alived[minionid] %}
+{%       set is_alive = True %}
+{%     endif %}
+{%   endif %}
+{%   if node_type not in node_types.keys() %}
+{%     do node_types.update({node_type: {hostname: {'ip':ip[0], 'alive':is_alive }}}) %}
+{%   else %}
+{%     if hostname not in node_types[node_type] %}
+{%       do node_types[node_type].update({hostname: {'ip':ip[0], 'alive':is_alive}}) %}
+{%     else %}
+{%       do node_types[node_type][hostname].update({'ip':ip[0], 'alive':is_alive}) %}
+{%     endif %}
+{%   endif %}
+{% endfor %}
+
+node_data:
+{% for node_type, host_values in node_types.items() %}
+  {{node_type}}:
+{%   for hostname, details in host_values.items() %}
+    {{hostname}}:
+      ip: {{details.ip}}
+      alive: {{ details.alive }}
+{%   endfor %}
+{% endfor %}

pillar/top.sls

@@ -3,6 +3,9 @@ base:
     - patch.needs_restarting
     - logrotate
 
+  '* and not *_eval and not *_import':
+    - logstash.nodes
+
   '*_eval or *_helixsensor or *_heavynode or *_sensor or *_standalone or *_import':
     - match: compound
     - zeek
@@ -24,6 +27,9 @@ base:
     - data.*
 {% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
     - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
 {% endif %}
     - secrets
     - global
@@ -43,6 +49,9 @@ base:
     - elasticsearch.eval
 {% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
     - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
 {% endif %}
     - global
     - minions.{{ grains.id }}
@@ -54,6 +63,9 @@ base:
     - elasticsearch.search
 {% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
     - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
 {% endif %}
     - data.*
     - zeeklogs
@@ -95,12 +107,22 @@ base:
     - minions.{{ grains.id }}
     - data.nodestab
 
+  '*_receiver':
+    - logstash
+    - logstash.receiver
+    - elasticsearch.auth
+    - global
+    - minions.{{ grains.id }}
+
   '*_import':
     - zeeklogs
     - secrets
     - elasticsearch.eval
 {% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/elasticsearch/auth.sls') %}
     - elasticsearch.auth
+{% endif %}
+{% if salt['file.file_exists']('/opt/so/saltstack/local/pillar/kibana/secrets.sls') %}
+    - kibana.secrets
 {% endif %}
     - global
     - minions.{{ grains.id }}

salt/allowed_states.map.jinja

@@ -35,6 +35,7 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
@@ -45,10 +46,10 @@
           'schedule',
           'soctopus',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-heavynode': [
-          'ca',
           'ssl',
           'nginx',
           'telegraf',
@@ -78,7 +79,6 @@
           'docker_clean'
           ],
       'so-fleet': [
-          'ca',
           'ssl',
           'nginx',
           'telegraf',
@@ -99,6 +99,7 @@
           'manager',
           'nginx',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
@@ -108,7 +109,8 @@
           'zeek',
           'schedule',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-manager': [
           'salt.master',
@@ -121,13 +123,15 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
           'utility',
           'schedule',
           'soctopus',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-managersearch': [
           'salt.master',
@@ -139,6 +143,7 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'manager',
           'idstools',
@@ -146,10 +151,10 @@
           'utility',
           'schedule',
           'soctopus',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-node': [
-          'ca',
           'ssl',
           'nginx',
           'telegraf',
@@ -168,6 +173,7 @@
           'influxdb',
           'grafana',
           'soc',
+          'kratos',
           'firewall',
           'idstools',
           'suricata.manager',
@@ -178,10 +184,10 @@
           'schedule',
           'soctopus',
           'tcpreplay',
-          'docker_clean'
+          'docker_clean',
+          'learn'
           ],
       'so-sensor': [
-          'ca',
           'ssl',
           'telegraf',
           'firewall',
@@ -195,9 +201,16 @@
           'tcpreplay',
           'docker_clean'
           ],
+      'so-receiver': [
+          'ssl',
+          'telegraf',
+          'firewall',
+          'schedule',
+          'docker_clean'
+          ],
   }, grain='role') %}
 
-  {% if FILEBEAT and grains.role in ['so-helixsensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-import'] %}
+  {% if FILEBEAT and grains.role in ['so-helixsensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-import', 'so-receiver'] %}
     {% do allowed_states.append('filebeat') %}
   {% endif %}
 
@@ -205,7 +218,7 @@
     {% do allowed_states.append('mysql') %}
   {% endif %}
 
-  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-sensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+  {% if (FLEETMANAGER or FLEETNODE) and grains.role in ['so-sensor', 'so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-receiver'] %}
     {% do allowed_states.append('fleet.install_package') %}
   {% endif %}
 
@@ -225,7 +238,7 @@
     {% do allowed_states.append('strelka') %}
   {% endif %}
 
-  {% if WAZUH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode']%}
+  {% if WAZUH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-receiver']%}
     {% do allowed_states.append('wazuh') %}
   {% endif %}
 
@@ -233,11 +246,16 @@
     {% do allowed_states.append('elasticsearch') %}
   {% endif %}
 
-  {% if KIBANA and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
-    {% do allowed_states.append('kibana') %}
+  {% if ELASTICSEARCH and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    {% do allowed_states.append('elasticsearch.auth') %}
   {% endif %}
 
-  {% if CURATOR and grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+  {% if KIBANA and grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    {% do allowed_states.append('kibana') %}
+    {% do allowed_states.append('kibana.secrets') %}
+  {% endif %}
+
+  {% if grains.role in ['so-eval', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-manager'] %}
     {% do allowed_states.append('curator') %}
   {% endif %}
 
@@ -265,11 +283,11 @@
     {% do allowed_states.append('domainstats') %}
   {% endif %}
 
-  {% if LOGSTASH and grains.role in ['so-helixsensor', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode'] %}
+  {% if LOGSTASH and grains.role in ['so-helixsensor', 'so-manager', 'so-standalone', 'so-node', 'so-managersearch', 'so-heavynode', 'so-receiver'] %}
     {% do allowed_states.append('logstash') %}
   {% endif %}
 
-  {% if REDIS and grains.role in ['so-manager', 'so-standalone', 'so-managersearch', 'so-heavynode'] %}
+  {% if REDIS and grains.role in ['so-manager', 'so-standalone', 'so-managersearch', 'so-heavynode', 'so-receiver'] %}
     {% do allowed_states.append('redis') %}
   {% endif %}
 
@@ -296,4 +314,4 @@
 {% endif %}
 
 {# all nodes can always run salt.minion state #}
-{% do allowed_states.append('salt.minion') %}
+{% do allowed_states.append('salt.minion') %}

salt/ca/dirs.sls

@@ -0,0 +1,4 @@
+pki_issued_certs:
+  file.directory:
+    - name: /etc/pki/issued_certs
+    - makedirs: True

salt/ca/files/signing_policies.conf

@@ -1,3 +1,6 @@
+mine_functions:
+  x509.get_pem_entries: [/etc/pki/ca.crt]
+
 x509_signing_policies:
   filebeat:
     - minions: '*'

salt/ca/init.sls

@@ -1,17 +1,14 @@
 {% from 'allowed_states.map.jinja' import allowed_states %}
 {% if sls in allowed_states %}
 
+include:
+  - ca.dirs
+
 {% set manager = salt['grains.get']('master') %}
 /etc/salt/minion.d/signing_policies.conf:
   file.managed:
     - source: salt://ca/files/signing_policies.conf
 
-/etc/pki:
-  file.directory: []
-
-/etc/pki/issued_certs:
-  file.directory: []
-
 pki_private_key:
   x509.private_key_managed:
     - name: /etc/pki/ca.key
@@ -24,8 +21,9 @@ pki_private_key:
       - x509: /etc/pki/ca.crt
     {%- endif %}
 
-/etc/pki/ca.crt:
+pki_public_ca_crt:
   x509.certificate_managed:
+    - name: /etc/pki/ca.crt
     - signing_private_key: /etc/pki/ca.key
     - CN: {{ manager }}
     - C: US
@@ -41,18 +39,12 @@ pki_private_key:
     - backup: True
     - replace: False
     - require:
-      - file: /etc/pki
+      - sls: ca.dirs
     - timeout: 30
     - retry:
         attempts: 5
         interval: 30
 
-x509_pem_entries:
-  module.run:
-    - mine.send:
-       - name: x509.get_pem_entries
-       - glob_path: /etc/pki/ca.crt
-
 cakeyperms:
   file.managed:
     - replace: False
@@ -66,4 +58,4 @@ cakeyperms:
   test.fail_without_changes:
     - name: {{sls}}_state_not_allowed
 
-{% endif %}
+{% endif %}

salt/ca/remove.sls

@@ -0,0 +1,7 @@
+pki_private_key:
+  file.absent:
+    - name: /etc/pki/ca.key
+
+pki_public_ca_crt:
+  file.absent:
+    - name: /etc/pki/ca.crt

salt/common/files/log-rotate.conf

@@ -22,6 +22,7 @@
 /opt/so/log/salt/so-salt-minion-check
 /opt/so/log/salt/minion
 /opt/so/log/salt/master
+/opt/so/log/logscan/*.log
 {
     {{ logrotate_conf | indent(width=4) }}
 }

salt/common/init.sls

@@ -4,11 +4,22 @@
 {% set role = grains.id.split('_') | last %}
 {% from 'elasticsearch/auth.map.jinja' import ELASTICAUTH with context %}
 
+include:
+  - common.soup_scripts
+{% if grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+  - manager.elasticsearch # needed for elastic_curl_config state
+{% endif %}
+
 # Remove variables.txt from /tmp - This is temp
 rmvariablesfile:
   file.absent:
     - name: /tmp/variables.txt
 
+dockergroup:
+  group.present:
+    - name: docker
+    - gid: 920
+
 # Add socore Group
 socoregroup:
   group.present:
@@ -101,16 +112,24 @@ commonpkgs:
       - python3-m2crypto
       - python3-mysqldb
       - python3-packaging
+      - python3-lxml
       - git
       - vim
 
 heldpackages:
   pkg.installed:
     - pkgs:
+    {% if grains['oscodename'] == 'bionic' %}
       - containerd.io: 1.4.4-1
       - docker-ce: 5:20.10.5~3-0~ubuntu-bionic
       - docker-ce-cli: 5:20.10.5~3-0~ubuntu-bionic
       - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-bionic
+    {% elif grains['oscodename'] == 'focal' %}
+      - containerd.io: 1.4.9-1
+      - docker-ce: 5:20.10.8~3-0~ubuntu-focal
+      - docker-ce-cli: 5:20.10.5~3-0~ubuntu-focal
+      - docker-ce-rootless-extras: 5:20.10.5~3-0~ubuntu-focal
+    {% endif %}
     - hold: True
     - update_holds: True
 
@@ -136,6 +155,7 @@ commonpkgs:
       - python36-m2crypto
       - python36-mysql
       - python36-packaging
+      - python36-lxml
       - yum-utils
       - device-mapper-persistent-data
       - lvm2
@@ -168,6 +188,7 @@ alwaysupdated:
 Etc/UTC:
   timezone.system
 
+{% if salt['pillar.get']('elasticsearch:auth:enabled', False) %}
 elastic_curl_config:
   file.managed:
     - name: /opt/so/conf/elasticsearch/curl.config
@@ -175,6 +196,11 @@ elastic_curl_config:
     - mode: 600
     - show_changes: False
     - makedirs: True
+  {% if grains.role in ['so-eval', 'so-manager', 'so-standalone', 'so-managersearch', 'so-import'] %}
+    - require:
+      - file: elastic_curl_config_distributed
+  {% endif %}
+{% endif %}
 
 # Sync some Utilities
 utilsyncscripts:
@@ -189,6 +215,11 @@ utilsyncscripts:
         ELASTICCURL: 'curl'
     - context:
         ELASTICCURL: {{ ELASTICAUTH.elasticcurl }}
+    - exclude_pat:
+        - so-common
+        - so-firewall
+        - so-image-common
+        - soup
 
 {% if role in ['eval', 'standalone', 'sensor', 'heavynode'] %}
 # Add sensor cleanup
@@ -326,6 +357,16 @@ dockerreserveports:
     - name: /etc/sysctl.d/99-reserved-ports.conf
 
 {% if salt['grains.get']('sosmodel', '') %}
+  {% if grains['os'] == 'CentOS' %}     
+# Install Raid tools
+raidpkgs:
+  pkg.installed:
+    - skip_suggestions: True
+    - pkgs:
+      - securityonion-raidtools
+      - securityonion-megactl
+  {% endif %}
+
 # Install raid check cron
 /usr/sbin/so-raid-status > /dev/null 2>&1:
   cron.present:

salt/common/soup_scripts.sls

@@ -0,0 +1,13 @@
+# Sync some Utilities
+soup_scripts:
+  file.recurse:
+    - name: /usr/sbin
+    - user: root
+    - group: root
+    - file_mode: 755
+    - source: salt://common/tools/sbin
+    - include_pat:
+        - so-common
+        - so-firewall
+        - so-image-common
+        - soup

salt/common/tools/sbin/so-allow

@@ -1,6 +1,6 @@
-#!/bin/bash
+#!/usr/bin/env python3
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -15,152 +15,193 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-. /usr/sbin/so-common
+import ipaddress
+import textwrap
+import os
+import subprocess
+import sys
+import argparse
+import re
+from lxml import etree as ET
+from datetime import datetime as dt
+from datetime import timezone as tz
 
-local_salt_dir=/opt/so/saltstack/local
-
-SKIP=0
-
-function usage {
-
-cat << EOF
-
-Usage: $0 [-abefhoprsw] [ -i IP ]
-
-This program allows you to add a firewall rule to allow connections from a new IP address or CIDR range.
-
-If you run this program with no arguments, it will present a menu for you to choose your options.
-
-If you want to automate and skip the menu, you can pass the desired options as command line arguments.
-
-EXAMPLES
-
-To add 10.1.2.3 to the analyst role:
-so-allow -a -i 10.1.2.3
-
-To add 10.1.2.0/24 to the osquery role:
-so-allow -o -i 10.1.2.0/24
-
-EOF
 
+LOCAL_SALT_DIR='/opt/so/saltstack/local'
+WAZUH_CONF='/nsm/wazuh/etc/ossec.conf'
+VALID_ROLES = {
+  'a': { 'role': 'analyst','desc': 'Analyst - 80/tcp, 443/tcp' },
+  'b': { 'role': 'beats_endpoint', 'desc': 'Logstash Beat - 5044/tcp' },
+  'e': { 'role': 'elasticsearch_rest', 'desc': 'Elasticsearch REST API - 9200/tcp' },
+  'f': { 'role': 'strelka_frontend', 'desc': 'Strelka frontend - 57314/tcp' },
+  'o': { 'role': 'osquery_endpoint', 'desc': 'Osquery endpoint - 8090/tcp' },
+  's': { 'role': 'syslog', 'desc': 'Syslog device - 514/tcp/udp' },
+  'w': { 'role': 'wazuh_agent', 'desc': 'Wazuh agent - 1514/tcp/udp' },
+  'p': { 'role': 'wazuh_api', 'desc': 'Wazuh API - 55000/tcp' },
+  'r': { 'role': 'wazuh_authd', 'desc': 'Wazuh registration service - 1515/tcp' }
 }
 
-while getopts "ahfesprbowi:" OPTION
-do
-    case $OPTION in
-        h)
-            usage
-            exit 0
-            ;;
-        a)
-            FULLROLE="analyst"
-            SKIP=1
-            ;;
-        b)
-            FULLROLE="beats_endpoint"
-            SKIP=1
-            ;;
-	e)
-            FULLROLE="elasticsearch_rest"
-            SKIP=1
-            ;;
-        f)
-	    FULLROLE="strelka_frontend"
-            SKIP=1
-            ;;
-        i)  IP=$OPTARG
-            ;;
-        o)
-            FULLROLE="osquery_endpoint"
-            SKIP=1
-            ;;
-        w)
-            FULLROLE="wazuh_agent"
-            SKIP=1
-            ;;
-        s)
-            FULLROLE="syslog"
-            SKIP=1
-            ;;
-        p)
-            FULLROLE="wazuh_api"
-            SKIP=1
-            ;;
-        r)
-            FULLROLE="wazuh_authd"
-            SKIP=1
-            ;;
-        *)
-            usage
-            exit 0
-            ;;
-    esac
-done
 
-if [ "$SKIP" -eq 0 ]; then
+def validate_ip_cidr(ip_cidr: str) -> bool:
+  try:
+    ipaddress.ip_address(ip_cidr)
+  except ValueError:
+    try:
+      ipaddress.ip_network(ip_cidr)
+    except ValueError:
+      return False
+  return True
 
-    echo "This program allows you to add a firewall rule to allow connections from a new IP address."
-    echo ""
-    echo "Choose the role for the IP or Range you would like to add"
-    echo ""
-    echo "[a] - Analyst - ports 80/tcp and 443/tcp"
-    echo "[b] - Logstash Beat - port 5044/tcp"
-    echo "[e] - Elasticsearch REST API - port 9200/tcp"
-    echo "[f] - Strelka frontend - port 57314/tcp"
-    echo "[o] - Osquery endpoint - port 8090/tcp"
-    echo "[s] - Syslog device - 514/tcp/udp"
-    echo "[w] - Wazuh agent - port 1514/tcp/udp"
-    echo "[p] - Wazuh API - port 55000/tcp"
-    echo "[r] - Wazuh registration service - 1515/tcp"
-    echo ""
-    echo "Please enter your selection:"
-    read -r ROLE
-    echo "Enter a single ip address or range to allow (example: 10.10.10.10 or 10.10.0.0/16):"
-    read -r IP
 
-    if [ "$ROLE" == "a" ]; then
-        FULLROLE=analyst
-    elif [ "$ROLE" == "b" ]; then
-        FULLROLE=beats_endpoint
-    elif [ "$ROLE" == "e" ]; then
-        FULLROLE=elasticsearch_rest
-    elif [ "$ROLE" == "f" ]; then
-        FULLROLE=strelka_frontend
-    elif [ "$ROLE" == "o" ]; then
-        FULLROLE=osquery_endpoint
-    elif [ "$ROLE" == "w" ]; then
-        FULLROLE=wazuh_agent
-    elif [ "$ROLE" == "s" ]; then
-        FULLROLE=syslog
-    elif [ "$ROLE" == "p" ]; then
-        FULLROLE=wazuh_api
-    elif [ "$ROLE" == "r" ]; then
-        FULLROLE=wazuh_authd
-    else
-        echo "I don't recognize that role"
-        exit 1
-    fi
+def role_prompt() -> str:
+  print()
+  print('Choose the role for the IP or Range you would like to allow')
+  print()
+  for role in VALID_ROLES:
+    print(f'[{role}] - {VALID_ROLES[role]["desc"]}')
+  print()
+  role = input('Please enter your selection: ')
+  if role in VALID_ROLES.keys():
+    return VALID_ROLES[role]['role']
+  else:
+    print(f'Invalid role \'{role}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+  
 
-fi
+def ip_prompt() -> str:
+  ip = input('Enter a single ip address or range to allow (ex: 10.10.10.10 or 10.10.0.0/16): ')
+  if validate_ip_cidr(ip):
+    return ip
+  else:
+    print(f'Invalid IP address or CIDR block \'{ip}\', please try again.', file=sys.stderr)
+    sys.exit(1)
 
-echo "Adding $IP to the $FULLROLE role. This can take a few seconds"
-/usr/sbin/so-firewall includehost $FULLROLE $IP
-salt-call state.apply firewall queue=True
 
-# Check if Wazuh enabled
-if grep -q -R "wazuh: 1" $local_salt_dir/pillar/*; then
-    # If analyst, add to Wazuh AR whitelist
-    if [ "$FULLROLE" == "analyst" ]; then
-        WAZUH_MGR_CFG="/nsm/wazuh/etc/ossec.conf"
-        if ! grep -q "<white_list>$IP</white_list>" $WAZUH_MGR_CFG ; then
-            DATE=$(date)
-            sed -i 's/<\/ossec_config>//' $WAZUH_MGR_CFG
-            sed -i '/^$/N;/^\n$/D' $WAZUH_MGR_CFG
-            echo -e "<!--Address $IP added by /usr/sbin/so-allow on \"$DATE\"-->\n  <global>\n    <white_list>$IP</white_list>\n  </global>\n</ossec_config>" >> $WAZUH_MGR_CFG
-            echo "Added whitelist entry for $IP in $WAZUH_MGR_CFG."
-        echo
-            echo "Restarting OSSEC Server..."
-            /usr/sbin/so-wazuh-restart
-        fi
-    fi
-fi
+def wazuh_enabled() -> bool:
+  file = f'{LOCAL_SALT_DIR}/pillar/global.sls'
+  with open(file, 'r') as pillar:
+    if 'wazuh: 1' in pillar.read():
+      return True
+  return False
+
+
+def root_to_str(root: ET.ElementTree) -> str:
+    return ET.tostring(root, encoding='unicode', method='xml', xml_declaration=False, pretty_print=True)
+
+
+def add_wl(ip):
+    parser = ET.XMLParser(remove_blank_text=True)
+    with open(WAZUH_CONF, 'rb') as wazuh_conf:
+        tree = ET.parse(wazuh_conf, parser)
+    root = tree.getroot()
+
+    source_comment = ET.Comment(f'Address {ip} added by /usr/sbin/so-allow on {dt.utcnow().replace(tzinfo=tz.utc).strftime("%a %b %e %H:%M:%S %Z %Y")}')
+    new_global = ET.Element("global")
+    new_wl = ET.SubElement(new_global, 'white_list')
+    new_wl.text = ip
+
+    root.append(source_comment)
+    root.append(new_global)
+
+    with open(WAZUH_CONF, 'w') as add_out:
+        add_out.write(root_to_str(root))
+
+
+def apply(role: str, ip: str) -> int:
+  firewall_cmd = ['so-firewall', 'includehost', role, ip]
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'firewall', 'queue=True']
+  restart_wazuh_cmd = ['so-wazuh-restart']
+  print(f'Adding {ip} to the {role} role. This can take a few seconds...')
+  cmd = subprocess.run(firewall_cmd)
+  if cmd.returncode == 0:
+    cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  else:
+    return cmd.returncode
+  if cmd.returncode == 0:
+    if wazuh_enabled() and role=='analyst':
+      try:
+        add_wl(ip)
+        print(f'Added whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+      except Exception as e:
+        print(f'Failed to add whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+        print(e)
+        return 1
+      print('Restarting OSSEC Server...')
+      cmd = subprocess.run(restart_wazuh_cmd)
+    else:
+      return cmd.returncode
+  else:
+    print(f'Commmand \'{" ".join(salt_cmd)}\' failed.', file=sys.stderr)
+    return cmd.returncode
+  if cmd.returncode != 0:
+    print('Failed to restart OSSEC server.')
+  return cmd.returncode
+
+
+def main():
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=textwrap.dedent(f'''\
+        additional information:
+                      To use this script in interactive mode call it with no arguments
+        '''
+  ))
+
+  group = main_parser.add_argument_group(title='roles')
+  group.add_argument('-a', dest='roles', action='append_const', const=VALID_ROLES['a']['role'], help="Analyst - 80/tcp, 443/tcp")
+  group.add_argument('-b', dest='roles', action='append_const', const=VALID_ROLES['b']['role'], help="Logstash Beat - 5044/tcp")
+  group.add_argument('-e', dest='roles', action='append_const', const=VALID_ROLES['e']['role'], help="Elasticsearch REST API - 9200/tcp")
+  group.add_argument('-f', dest='roles', action='append_const', const=VALID_ROLES['f']['role'], help="Strelka frontend - 57314/tcp")
+  group.add_argument('-o', dest='roles', action='append_const', const=VALID_ROLES['o']['role'], help="Osquery endpoint - 8090/tcp")
+  group.add_argument('-s', dest='roles', action='append_const', const=VALID_ROLES['s']['role'], help="Syslog device - 514/tcp/udp")
+  group.add_argument('-w', dest='roles', action='append_const', const=VALID_ROLES['w']['role'], help="Wazuh agent - 1514/tcp/udp")
+  group.add_argument('-p', dest='roles', action='append_const', const=VALID_ROLES['p']['role'], help="Wazuh API - 55000/tcp")
+  group.add_argument('-r', dest='roles', action='append_const', const=VALID_ROLES['r']['role'], help="Wazuh registration service - 1515/tcp")
+  
+  ip_g = main_parser.add_argument_group(title='allow')
+  ip_g.add_argument('-i', help="IP or CIDR block to disallow connections from, requires at least one role argument", metavar='', dest='ip')
+
+  args = main_parser.parse_args(sys.argv[1:])
+
+  if args.roles is None:
+    role = role_prompt()
+    ip = ip_prompt()
+    try:
+      return_code = apply(role, ip)
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+    sys.exit(return_code)
+  elif args.roles is not None and args.ip is None:
+    if os.environ.get('IP') is None:
+      main_parser.print_help()
+      sys.exit(1)
+    else:
+      args.ip = os.environ['IP']
+  
+  if validate_ip_cidr(args.ip):
+    try:
+      for role in args.roles:
+        return_code = apply(role, args.ip)
+        if return_code > 0:
+          break
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+  else:
+    print(f'Invalid IP address or CIDR block \'{args.ip}\', please try again.', file=sys.stderr)
+    return_code = 1
+    
+  sys.exit(return_code)
+
+
+if __name__ == '__main__':
+  try:
+    main()
+  except KeyboardInterrupt:
+    sys.exit(1)
+

salt/common/tools/sbin/so-allow-view

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -20,4 +20,4 @@
 echo ""
 echo "Hosts/Networks that have access to login to the Security Onion Console:"
 
-so-firewall includedhosts analyst
+so-firewall includedhosts analyst

salt/common/tools/sbin/so-analyst-install

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014-2020 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 
 #    This program is free software: you can redistribute it and/or modify
 #    it under the terms of the GNU General Public License as published by
@@ -108,7 +108,7 @@ CANCURL=$(curl -sI https://securityonionsolutions.com/ | grep "200 OK")
     while [[ $CURLCONTINUE != "yes" ]] && [[ $CURLCONTINUE != "no" ]]; do
       if [[ "$FIRSTPASS" == "yes" ]]; then
         echo "We could not access https://securityonionsolutions.com/."
-        echo "Since packages are downloaded from the internet, internet acceess is required."
+        echo "Since packages are downloaded from the internet, internet access is required."
         echo "If you would like to ignore this warning and continue anyway, please type 'yes'."
         echo "Otherwise, type 'no' to exit."
         FIRSTPASS=no

salt/common/tools/sbin/so-checkin

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-salt-call state.highstate -linfo 
+salt-call state.highstate -l info 

salt/common/tools/sbin/so-common

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -99,6 +99,15 @@ check_password() {
 	return $?
 }
 
+check_password_and_exit() {
+	local password=$1
+	if ! check_password "$password"; then
+		echo "Password is invalid. Do not include single quotes, double quotes, dollar signs, and backslashes in the password."
+		exit 2
+	fi
+	return 0
+}
+
 check_elastic_license() {
 
 	[ -n "$TESTING" ] && return
@@ -285,32 +294,49 @@ require_manager() {
 }
 
 retry() {
-	maxAttempts=$1
-	sleepDelay=$2
-	cmd=$3
-	expectedOutput=$4
-	attempt=0
-	local exitcode=0
-	while [[ $attempt -lt $maxAttempts ]]; do			
-		attempt=$((attempt+1))
-		echo "Executing command with retry support: $cmd"
-		output=$(eval "$cmd")
-		exitcode=$?
-		echo "Results: $output ($exitcode)"
-		if [ -n "$expectedOutput" ]; then
-			if [[ "$output" =~ "$expectedOutput" ]]; then
-				return $exitCode
-			else
-				echo "Expected '$expectedOutput' but got '$output'"
-			fi
-		elif [[ $exitcode -eq 0 ]]; then
-			return $exitCode
-		fi
-		echo "Command failed with exit code $exitcode; will retry in $sleepDelay seconds ($attempt / $maxAttempts)..."
-		sleep $sleepDelay
-	done
-	echo "Command continues to fail; giving up."
-	return $exitcode
+        maxAttempts=$1
+        sleepDelay=$2
+        cmd=$3
+        expectedOutput=$4
+        failedOutput=$5
+        attempt=0
+        local exitcode=0
+        while [[ $attempt -lt $maxAttempts ]]; do
+                attempt=$((attempt+1))
+                echo "Executing command with retry support: $cmd"
+                output=$(eval "$cmd")
+                exitcode=$?
+                echo "Results: $output ($exitcode)"
+                if [ -n "$expectedOutput" ]; then
+                        if [[ "$output" =~ "$expectedOutput" ]]; then
+                                return $exitcode
+                        else
+                                echo "Did not find expectedOutput: '$expectedOutput' in the output below from running the command: '$cmd'"
+								echo "<Start of output>"
+								echo "$output"
+								echo "<End of output>"
+                        fi
+                elif [ -n "$failedOutput" ]; then
+                        if [[ "$output" =~ "$failedOutput" ]]; then
+                                echo "Found failedOutput: '$failedOutput' in the output below from running the command: '$cmd'"
+								echo "<Start of output>"
+								echo "$output"
+								echo "<End of output>"
+								if [[ $exitcode -eq 0 ]]; then
+									echo "The exitcode was 0, but we are setting to 1 since we found $failedOutput in the output."
+									exitcode=1
+								fi
+                        else
+                                return $exitcode
+                        fi
+                elif [[ $exitcode -eq 0 ]]; then
+                        return $exitcode
+                fi
+                echo "Command failed with exit code $exitcode; will retry in $sleepDelay seconds ($attempt / $maxAttempts)..."
+                sleep $sleepDelay
+        done
+        echo "Command continues to fail; giving up."
+        return $exitcode
 }
 
 run_check_net_err() {
@@ -334,6 +360,13 @@ run_check_net_err() {
 		exit $exit_code
 	fi
 }
+set_cron_service_name() {
+  if [[ "$OS" == "centos" ]]; then
+    cron_service_name="crond"
+  else
+    cron_service_name="cron"
+  fi
+}
 
 set_os() {
 	if [ -f /etc/redhat-release ]; then
@@ -372,18 +405,44 @@ set_version() {
 	fi
 }
 
+systemctl_func() {
+	local action=$1
+	local echo_action=$1
+	local service_name=$2
+
+	if [[ "$echo_action" == "stop" ]]; then
+		echo_action="stopp"
+	fi
+
+	echo ""
+	echo "${echo_action^}ing $service_name service at $(date +"%T.%6N")"
+    systemctl $action $service_name && echo "Successfully ${echo_action}ed $service_name." || echo "Failed to $action $service_name."
+	echo ""
+}
+
+has_uppercase() {
+	local string=$1
+
+	echo "$string" | grep -qP '[A-Z]' \
+		&& return 0 \
+		|| return 1
+}
+
 valid_cidr() {
 	# Verify there is a backslash in the string
 	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
-	
-	local cidr
-	local ip
 
-	cidr=$(echo "$1" | sed 's/.*\///')
-	ip=$(echo "$1" | sed 's/\/.*//' )
+	valid_ip4_cidr_mask "$1" && return 0 || return 1
+	
+	local cidr="$1"
+	local ip
+	ip=$(echo "$cidr" | sed 's/\/.*//' )
 	
 	if valid_ip4 "$ip"; then
-		[[ $cidr =~ ([0-9]|[1-2][0-9]|3[0-2]) ]] && return 0 || return 1
+		local ip1 ip2 ip3 ip4 N
+		IFS="./" read -r ip1 ip2 ip3 ip4 N <<< "$cidr"
+		ip_total=$((ip1 * 256 ** 3 + ip2 * 256 ** 2 + ip3 * 256 + ip4))
+		[[ $((ip_total % 2**(32-N))) == 0 ]] && return 0 || return 1
 	else
 		return 1
 	fi
@@ -433,6 +492,23 @@ valid_ip4() {
 	echo "$ip" | grep -qP '^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$' && return 0 || return 1
 }
 
+valid_ip4_cidr_mask() {
+	# Verify there is a backslash in the string
+	echo "$1" | grep -qP "^[^/]+/[^/]+$" || return 1
+
+	local cidr
+	local ip
+
+	cidr=$(echo "$1" | sed 's/.*\///')
+	ip=$(echo "$1" | sed 's/\/.*//' )
+
+	if valid_ip4 "$ip"; then
+		[[ $cidr =~ ^([0-9]|[1-2][0-9]|3[0-2])$ ]] && return 0 || return 1
+	else
+		return 1
+	fi
+}
+
 valid_int() {
 	local num=$1
 	local min=${2:-1}

salt/common/tools/sbin/so-config-backup

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -45,4 +45,4 @@ while [ "$NUMBACKUPS" -gt "$MAXBACKUPS" ]; do
   OLDESTBACKUP=$(find /nsm/backup/ -type f -name "so-config-backup*" -type f -printf '%T+ %p\n' | sort | head -n 1 | awk -F" " '{print $2}')
   rm -f $OLDESTBACKUP
   NUMBACKUPS=$(find /nsm/backup/ -type f -name "so-config-backup*" | wc -l)
-done
+done

salt/common/tools/sbin/so-cortex-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-cortex-user-add

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -51,4 +51,4 @@ else
     echo $resp
     exit 2
 fi
-    
+    

salt/common/tools/sbin/so-cortex-user-enable

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -54,4 +54,4 @@ else
     echo $resp
     exit 2
 fi
-    
+    

salt/common/tools/sbin/so-curator-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-curator-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-curator-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-deny

@@ -0,0 +1,213 @@
+#!/usr/bin/env python3
+
+# Copyright 2014-2022 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import ipaddress
+import textwrap
+import os
+import subprocess
+import sys
+import argparse
+import re
+from lxml import etree as ET
+from xml.dom import minidom
+
+
+LOCAL_SALT_DIR='/opt/so/saltstack/local'
+WAZUH_CONF='/nsm/wazuh/etc/ossec.conf'
+VALID_ROLES = {
+  'a': { 'role': 'analyst','desc': 'Analyst - 80/tcp, 443/tcp' },
+  'b': { 'role': 'beats_endpoint', 'desc': 'Logstash Beat - 5044/tcp' },
+  'e': { 'role': 'elasticsearch_rest', 'desc': 'Elasticsearch REST API - 9200/tcp' },
+  'f': { 'role': 'strelka_frontend', 'desc': 'Strelka frontend - 57314/tcp' },
+  'o': { 'role': 'osquery_endpoint', 'desc': 'Osquery endpoint - 8090/tcp' },
+  's': { 'role': 'syslog', 'desc': 'Syslog device - 514/tcp/udp' },
+  'w': { 'role': 'wazuh_agent', 'desc': 'Wazuh agent - 1514/tcp/udp' },
+  'p': { 'role': 'wazuh_api', 'desc': 'Wazuh API - 55000/tcp' },
+  'r': { 'role': 'wazuh_authd', 'desc': 'Wazuh registration service - 1515/tcp' }
+}
+
+
+def validate_ip_cidr(ip_cidr: str) -> bool:
+  try:
+    ipaddress.ip_address(ip_cidr)
+  except ValueError:
+    try:
+      ipaddress.ip_network(ip_cidr)
+    except ValueError:
+      return False
+  return True
+
+
+def role_prompt() -> str:
+  print()
+  print('Choose the role for the IP or Range you would like to deny')
+  print()
+  for role in VALID_ROLES:
+    print(f'[{role}] - {VALID_ROLES[role]["desc"]}')
+  print()
+  role = input('Please enter your selection: ')
+  if role in VALID_ROLES.keys():
+    return VALID_ROLES[role]['role']
+  else:
+    print(f'Invalid role \'{role}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+  
+
+def ip_prompt() -> str:
+  ip = input('Enter a single ip address or range to deny (ex: 10.10.10.10 or 10.10.0.0/16): ')
+  if validate_ip_cidr(ip):
+    return ip
+  else:
+    print(f'Invalid IP address or CIDR block \'{ip}\', please try again.', file=sys.stderr)
+    sys.exit(1)
+
+
+def wazuh_enabled() -> bool:
+  for file in os.listdir(f'{LOCAL_SALT_DIR}/pillar'):
+    with open(file, 'r') as pillar:
+      if 'wazuh: 1' in pillar.read():
+        return True
+  return False
+
+
+def root_to_str(root: ET.ElementTree) -> str:
+    xml_str = ET.tostring(root, encoding='unicode', method='xml').replace('\n', '')
+    xml_str = re.sub(r'(?:(?<=>) *)', '', xml_str)
+
+    # Remove specific substrings to better format comments on intial parse/write
+    xml_str = re.sub(r'       -', '', xml_str)
+    xml_str = re.sub(r'      -->', ' -->', xml_str)
+    
+    dom = minidom.parseString(xml_str)
+    return dom.toprettyxml(indent="  ")
+
+
+def rem_wl(ip):
+    parser = ET.XMLParser(remove_blank_text=True)
+    with open(WAZUH_CONF, 'rb') as wazuh_conf:
+        tree = ET.parse(wazuh_conf, parser)
+    root = tree.getroot()
+
+    global_elems = root.findall(f"global/white_list[. = '{ip}']/..")
+    if len(global_elems) > 0:
+      for g_elem in global_elems:
+          ge_index = list(root).index(g_elem)
+          if ge_index > 0 and root[list(root).index(g_elem) - 1].tag == ET.Comment:
+              root.remove(root[ge_index - 1])
+          root.remove(g_elem)
+
+      with open(WAZUH_CONF, 'w') as out:
+          out.write(root_to_str(root))
+
+
+def apply(role: str, ip: str) -> int:
+  firewall_cmd = ['so-firewall', 'excludehost', role, ip]
+  salt_cmd = ['salt-call', 'state.apply', '-l', 'quiet', 'firewall', 'queue=True']
+  restart_wazuh_cmd = ['so-wazuh-restart']
+  print(f'Removing {ip} from the {role} role. This can take a few seconds...')
+  cmd = subprocess.run(firewall_cmd)
+  if cmd.returncode == 0:
+    cmd = subprocess.run(salt_cmd, stdout=subprocess.DEVNULL)
+  else:
+    return cmd.returncode
+  if cmd.returncode == 0:
+    if wazuh_enabled and role=='analyst':
+      try:
+        rem_wl(ip)
+        print(f'Removed whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+      except Exception as e:
+        print(f'Failed to remove whitelist entry for {ip} from {WAZUH_CONF}', file=sys.stderr)
+        print(e)
+        return 1
+      print('Restarting OSSEC Server...')
+      cmd = subprocess.run(restart_wazuh_cmd)
+    else:
+      return cmd.returncode
+  else:
+    print(f'Commmand \'{" ".join(salt_cmd)}\' failed.', file=sys.stderr)
+    return cmd.returncode
+  if cmd.returncode != 0:
+    print('Failed to restart OSSEC server.')
+  return cmd.returncode
+
+
+def main():
+  if os.geteuid() != 0:
+    print('You must run this script as root', file=sys.stderr)
+    sys.exit(1)
+
+  main_parser = argparse.ArgumentParser(
+    formatter_class=argparse.RawDescriptionHelpFormatter,
+    epilog=textwrap.dedent(f'''\
+        additional information:
+                      To use this script in interactive mode call it with no arguments
+        '''
+  ))
+
+  group = main_parser.add_argument_group(title='roles')
+  group.add_argument('-a', dest='roles', action='append_const', const=VALID_ROLES['a']['role'], help="Analyst - 80/tcp, 443/tcp")
+  group.add_argument('-b', dest='roles', action='append_const', const=VALID_ROLES['b']['role'], help="Logstash Beat - 5044/tcp")
+  group.add_argument('-e', dest='roles', action='append_const', const=VALID_ROLES['e']['role'], help="Elasticsearch REST API - 9200/tcp")
+  group.add_argument('-f', dest='roles', action='append_const', const=VALID_ROLES['f']['role'], help="Strelka frontend - 57314/tcp")
+  group.add_argument('-o', dest='roles', action='append_const', const=VALID_ROLES['o']['role'], help="Osquery endpoint - 8090/tcp")
+  group.add_argument('-s', dest='roles', action='append_const', const=VALID_ROLES['s']['role'], help="Syslog device - 514/tcp/udp")
+  group.add_argument('-w', dest='roles', action='append_const', const=VALID_ROLES['w']['role'], help="Wazuh agent - 1514/tcp/udp")
+  group.add_argument('-p', dest='roles', action='append_const', const=VALID_ROLES['p']['role'], help="Wazuh API - 55000/tcp")
+  group.add_argument('-r', dest='roles', action='append_const', const=VALID_ROLES['r']['role'], help="Wazuh registration service - 1515/tcp")
+  
+  ip_g = main_parser.add_argument_group(title='allow')
+  ip_g.add_argument('-i', help="IP or CIDR block to disallow connections from, requires at least one role argument", metavar='', dest='ip')
+
+  args = main_parser.parse_args(sys.argv[1:])
+
+  if args.roles is None:
+    role = role_prompt()
+    ip = ip_prompt()
+    try:
+      return_code = apply(role, ip)
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+    sys.exit(return_code)
+  elif args.roles is not None and args.ip is None:
+    if os.environ.get('IP') is None:
+      main_parser.print_help()
+      sys.exit(1)
+    else:
+      args.ip = os.environ['IP']
+  
+  if validate_ip_cidr(args.ip):
+    try:
+      for role in args.roles:
+        return_code = apply(role, args.ip)
+        if return_code > 0:
+          break
+    except Exception as e:
+      print(f'Unexpected exception occurred: {e}', file=sys.stderr)
+      return_code = e.errno
+  else:
+    print(f'Invalid IP address or CIDR block \'{args.ip}\', please try again.', file=sys.stderr)
+    return_code = 1
+    
+  sys.exit(return_code)
+
+
+if __name__ == '__main__':
+  try:
+    main()
+  except KeyboardInterrupt:
+    sys.exit(1)

salt/common/tools/sbin/so-docker-prune

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-docker-refresh

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastalert-test

@@ -70,7 +70,7 @@ do
 done
 
 docker_exec(){
-	CMD="docker exec -it so-elastalert elastalert-test-rule /opt/elastalert/rules/$RULE_NAME --config /opt/config/elastalert_config.yaml $OPTIONS"
+	CMD="docker exec -it so-elastalert elastalert-test-rule /opt/elastalert/rules/$RULE_NAME --config /opt/elastalert/config.yaml $OPTIONS"
 	if [ "${RESULTS_TO_LOG,,}" = "y" ] ; then
 		$CMD > "$FILE_SAVE_LOCATION"
 	else

salt/common/tools/sbin/so-elastic-auth

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-auth-password-reset

@@ -0,0 +1,155 @@
+#!/bin/bash
+
+# Copyright 2014-2022 Security Onion Solutions, LLC
+
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+source $(dirname $0)/so-common
+require_manager
+
+user=$1
+elasticUsersFile=${ELASTIC_USERS_FILE:-/opt/so/saltstack/local/salt/elasticsearch/files/users}
+elasticAuthPillarFile=${ELASTIC_AUTH_PILLAR_FILE:-/opt/so/saltstack/local/pillar/elasticsearch/auth.sls}
+
+if [[ $# -ne 1 ]]; then
+  echo "Usage: $0 <user>"
+  echo ""
+  echo " where <user> is one of the following:"
+  echo ""
+  echo "         all: Reset the password for the so_elastic, so_kibana, so_logstash, so_beats, and so_monitor users"
+  echo "  so_elastic: Reset the password for the so_elastic user"
+  echo "   so_kibana: Reset the password for the so_kibana user"
+  echo " so_logstash: Reset the password for the so_logstash user"
+  echo "    so_beats: Reset the password for the so_beats user"
+  echo "  so_monitor: Reset the password for the so_monitor user"
+  echo ""
+  exit 1
+fi
+
+# function to create a lock so that the so-user sync cronjob can't run while this is running
+function lock() {
+  # Obtain file descriptor lock
+  exec 99>/var/tmp/so-user.lock || fail "Unable to create lock descriptor; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  flock -w 10 99 || fail "Another process is using so-user; if the system was not shutdown gracefully you may need to remove /var/tmp/so-user.lock manually."
+  trap 'rm -f /var/tmp/so-user.lock' EXIT
+}
+
+function unlock() {
+  rm -f /var/tmp/so-user.lock
+}
+
+function fail() {
+  msg=$1
+  echo "$1"
+  exit 1
+}
+
+function removeSingleUserPass() {
+  local user=$1
+  sed -i '/user: '"${user}"'/{N;/pass: /d}' "${elasticAuthPillarFile}"
+}
+
+function removeAllUserPass() {
+  local userList=("so_elastic" "so_kibana" "so_logstash" "so_beats" "so_monitor")
+
+  for u in ${userList[@]}; do
+    removeSingleUserPass "$u"
+  done
+}
+
+function removeElasticUsersFile() {
+  rm -f "$elasticUsersFile"
+}
+
+function createElasticAuthPillar() {
+  salt-call state.apply elasticsearch.auth queue=True
+}
+
+# this will disable highstate to prevent a highstate from starting while the script is running
+# will also disable salt.minion-state-apply-test allow so-salt-minion-check cronjob to restart salt-minion service incase
+function disableSaltStates() {
+  printf "\nDisabling salt.minion-state-apply-test and highstate from running.\n\n"
+  salt-call state.disable salt.minion-state-apply-test
+  salt-call state.disable highstate
+}
+
+function enableSaltStates() {
+  printf "\nEnabling salt.minion-state-apply-test and highstate.\n\n"
+  salt-call state.enable salt.minion-state-apply-test
+  salt-call state.enable highstate
+}
+
+function killAllSaltJobs() {
+  printf "\nKilling all running salt jobs.\n\n"
+  salt-call saltutil.kill_all_jobs
+}
+
+function soUserSync() {
+  # apply this state to update /opt/so/saltstack/local/salt/elasticsearch/curl.config on the manager
+  salt-call state.sls_id elastic_curl_config_distributed manager queue=True
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' saltutil.kill_all_jobs
+  # apply this state to get the curl.config
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.sls_id elastic_curl_config common queue=True
+  $(dirname $0)/so-user sync
+  printf "\nApplying logstash state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply logstash queue=True
+  printf "\nApplying filebeat state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode or G@role:so-sensor or G@role:so-fleet' state.apply filebeat queue=True
+  printf "\nApplying kibana state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch' state.apply kibana queue=True
+  printf "\nApplying curator state to the appropriate nodes.\n\n"
+  salt -C 'G@role:so-standalone or G@role:so-eval or G@role:so-import or G@role:so-manager or G@role:so-managersearch or G@role:so-node or G@role:so-heavynode' state.apply curator queue=True
+}
+
+function highstateManager() {
+  killAllSaltJobs
+  printf "\nRunning highstate on the manager to finalize password reset.\n\n"
+  salt-call state.highstate -linfo queue=True
+}
+
+case "${user}" in
+
+  so_elastic | so_kibana | so_logstash | so_beats | so_monitor)
+    lock
+    killAllSaltJobs
+    disableSaltStates
+    removeSingleUserPass "$user"
+    createElasticAuthPillar
+    removeElasticUsersFile
+    unlock
+    soUserSync
+    enableSaltStates
+    highstateManager
+    ;;
+
+  all)
+    lock
+    killAllSaltJobs
+    disableSaltStates
+    removeAllUserPass
+    createElasticAuthPillar
+    removeElasticUsersFile
+    unlock
+    soUserSync
+    enableSaltStates
+    highstateManager
+    ;;
+
+  *)
+    fail "Unsupported user: $user"
+    ;;
+
+esac
+
+exit 0

salt/common/tools/sbin/so-elastic-clear

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-diagnose

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elastic-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-indices-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-indices-rw

@@ -1,7 +1,7 @@
 #!/bin/bash
 #
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-pipeline-stats

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-pipeline-view

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-pipelines-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-query

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-shards-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-template-remove

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-template-view

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-elasticsearch-templates-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-filebeat-module-setup

@@ -1,5 +1,5 @@
 #!/bin/bash
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -54,7 +54,7 @@ PIPELINES=$({{ ELASTICCURL }} -sk https://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_
 if [[ "$PIPELINES" -lt 5 ]]; then
   echo "Setting up ingest pipeline(s)"
 
-  for MODULE in activemq apache auditd aws azure barracuda bluecoat cef checkpoint cisco coredns crowdstrike cyberark cylance elasticsearch envoyproxy f5 fortinet gcp google_workspace googlecloud gsuite haproxy ibmmq icinga iis imperva infoblox iptables juniper kafka kibana logstash microsoft misp mongodb mssql mysql nats netscout nginx o365 okta osquery panw postgresql rabbitmq radware redis santa snort snyk sonicwall sophos squid suricata system tomcat traefik zeek zscaler
+  for MODULE in activemq apache auditd aws azure barracuda bluecoat cef checkpoint cisco coredns crowdstrike cyberark cylance elasticsearch envoyproxy f5 fortinet gcp google_workspace googlecloud gsuite haproxy ibmmq icinga iis imperva infoblox iptables juniper kafka kibana logstash microsoft mongodb mssql mysql nats netscout nginx o365 okta osquery panw postgresql rabbitmq radware redis santa snort snyk sonicwall sophos squid suricata system threatintel tomcat traefik zeek zscaler
   do
     echo "Loading $MODULE"
     docker exec -i so-filebeat filebeat setup modules -pipelines -modules $MODULE -c $FB_MODULE_YML

salt/common/tools/sbin/so-filebeat-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-filebeat-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-filebeat-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-firewall

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -35,6 +35,7 @@ def showUsage(options, args):
   print('')
   print('  General commands:')
   print('    help           - Prints this usage information.')
+  print('    apply          - Apply the firewall state.')
   print('')
   print('  Host commands:')
   print('    listhostgroups - Lists the known host groups.')
@@ -66,11 +67,11 @@ def checkDefaultPortsOption(options):
 
 def checkApplyOption(options):
   if "--apply" in options:
-    return apply()
+    return apply(None, None)
 
 def loadYaml(filename):
   file = open(filename, "r")
-  return yaml.load(file.read())
+  return yaml.safe_load(file.read())
 
 def writeYaml(filename, content):
   file = open(filename, "w")
@@ -328,7 +329,7 @@ def removehost(options, args):
     code = checkApplyOption(options)
   return code
 
-def apply():
+def apply(options, args):
   proc = subprocess.run(['salt-call', 'state.apply', 'firewall', 'queue=True'])
   return proc.returncode
 
@@ -356,7 +357,8 @@ def main():
     "addport": addport,
     "removeport": removeport,
     "addhostgroup": addhostgroup,
-    "addportgroup": addportgroup
+    "addportgroup": addportgroup,
+    "apply": apply
   }
 
   code=1

salt/common/tools/sbin/so-fleet-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-fleet-setup

@@ -2,11 +2,16 @@
 
 #so-fleet-setup $FleetEmail $FleetPassword 
 
+. /usr/sbin/so-common
+
 if [[ $# -ne 2 ]] ; then
       echo "Username or Password was not set - exiting now."
       exit 1
 fi
 
+USER_EMAIL=$1
+USER_PW=$2
+
 # Checking to see if required containers are started...
 if [ ! "$(docker ps -q -f name=so-fleet)" ]; then
         echo "Starting Docker Containers..."
@@ -17,8 +22,16 @@ fi
 
 docker exec so-fleet fleetctl config set --address https://127.0.0.1:8080 --tls-skip-verify --url-prefix /fleet
 docker exec so-fleet bash -c 'while [[ "$(curl -s -o /dev/null --insecure -w ''%{http_code}'' https://127.0.0.1:8080/fleet)" != "301" ]]; do sleep 5; done'
-docker exec so-fleet fleetctl setup --email $1 --password $2
 
+# Create Security Onion Fleet Service Account + Setup Fleet
+FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
+FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+docker exec so-fleet fleetctl setup --email $FLEET_SA_EMAIL --password $FLEET_SA_PW --name SO_ServiceAccount --org-name SO
+
+# Create User Account
+echo "$USER_PW" | so-fleet-user-add "$USER_EMAIL"
+
+# Import Packs & Configs
 docker exec so-fleet fleetctl apply -f /packs/palantir/Fleet/Endpoints/MacOS/osquery.yaml
 docker exec so-fleet fleetctl apply -f /packs/palantir/Fleet/Endpoints/Windows/osquery.yaml
 docker exec so-fleet fleetctl apply -f /packs/so/so-default.yml

salt/common/tools/sbin/so-fleet-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-fleet-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-fleet-user-add

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -18,7 +18,7 @@
 . /usr/sbin/so-common
 
 usage() {
-    echo "Usage: $0 <new-user-name>"
+    echo "Usage: $0 <new-user-email>"
     echo ""
     echo "Adds a new user to Fleet. The new password will be read from STDIN."
     exit 1
@@ -28,37 +28,42 @@ if [ $# -ne 1 ]; then
   usage
 fi
 
-USER=$1
 
-MYSQL_PASS=$(lookup_pillar_secret mysql)
-FLEET_IP=$(lookup_pillar fleet_ip)
-FLEET_USER=$USER
+USER_EMAIL=$1
+FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
+FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+MYSQL_PW=$(lookup_pillar_secret mysql)
 
 # Read password for new user from stdin
 test -t 0
 if [[ $? == 0 ]]; then
   echo "Enter new password:"
 fi
-read -rs FLEET_PASS
+read -rs USER_PASS
 
-if ! check_password "$FLEET_PASS"; then
-  echo "Password is invalid. Please exclude single quotes, double quotes and backslashes from the password."
-  exit 2
-fi
+check_password_and_exit "$USER_PASS"
+
+# Config fleetctl & login with the SO Service Account
+CONFIG_OUTPUT=$(docker exec so-fleet fleetctl config set --address https://127.0.0.1:8080 --tls-skip-verify --url-prefix /fleet  2>&1 )
+SALOGIN_OUTPUT=$(docker exec so-fleet fleetctl login --email $FLEET_SA_EMAIL --password $FLEET_SA_PW 2>&1)
 
-FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_PASS'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
 if [[ $? -ne 0 ]]; then
-	echo "Failed to generate Fleet password hash"
-	exit 2
+    echo "Unable to add user to Fleet; Fleet Service account login failed"
+    echo "$SALOGIN_OUTPUT"
+    exit 2
 fi
 
-MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
-    "INSERT INTO users (password,salt,username,email,admin,enabled) VALUES ('$FLEET_HASH','','$FLEET_USER','$FLEET_USER',1,1)" 2>&1)
+# Create New User
+CREATE_OUTPUT=$(docker exec so-fleet fleetctl user create --email $USER_EMAIL --name $USER_EMAIL --password $USER_PASS --global-role admin 2>&1)
 
 if [[ $? -eq 0 ]]; then
     echo "Successfully added user to Fleet"
 else
     echo "Unable to add user to Fleet; user might already exist"
-    echo "$MYSQL_OUTPUT"
+    echo "$CREATE_OUTPUT"
     exit 2
-fi
+fi
+
+# Disable forced password reset
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PW fleet -e \
+"UPDATE users SET admin_forced_password_reset = 0 WHERE email = '$USER_EMAIL'" 2>&1)

salt/common/tools/sbin/so-fleet-user-delete

@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Copyright 2014-2022 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-email>"
+    echo ""
+    echo "Deletes a user in Fleet"
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER_EMAIL=$1
+FLEET_SA_EMAIL=$(lookup_pillar_secret fleet_sa_email)
+FLEET_SA_PW=$(lookup_pillar_secret fleet_sa_password)
+
+# Config fleetctl & login with the SO Service Account
+CONFIG_OUTPUT=$(docker exec so-fleet fleetctl config set --address https://127.0.0.1:8080 --tls-skip-verify --url-prefix /fleet  2>&1 )
+SALOGIN_OUTPUT=$(docker exec so-fleet fleetctl login --email $FLEET_SA_EMAIL --password $FLEET_SA_PW 2>&1)
+
+if [[ $? -ne 0 ]]; then
+    echo "Unable to delete user from Fleet; Fleet Service account login failed"
+    echo "$SALOGIN_OUTPUT"
+    exit 2
+fi
+
+# Delete User
+DELETE_OUTPUT=$(docker exec so-fleet fleetctl user delete --email $USER_EMAIL 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully deleted user from Fleet"
+else
+    echo "Unable to delete user from Fleet"
+    echo "$DELETE_OUTPUT"
+    exit 2
+fi
+
+

salt/common/tools/sbin/so-fleet-user-enable

@@ -1,58 +0,0 @@
-#!/bin/bash
-#
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-. /usr/sbin/so-common
-
-usage() {
-    echo "Usage: $0 <user-name>"
-    echo ""
-    echo "Enables or disables a user in Fleet"
-    exit 1
-}
-
-if [ $# -ne 2 ]; then
-  usage
-fi
-
-USER=$1
-
-MYSQL_PASS=$(lookup_pillar_secret mysql)
-FLEET_IP=$(lookup_pillar fleet_ip)
-FLEET_USER=$USER
-
-case "${2^^}" in
-    FALSE | NO | 0)
-        FLEET_STATUS=0
-        ;;
-    TRUE | YES | 1)
-        FLEET_STATUS=1
-        ;;
-    *)
-        usage
-        ;;
-esac
-
-MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
-    "UPDATE users SET enabled=$FLEET_STATUS WHERE username='$FLEET_USER'" 2>&1)
-
-if [[ $? -eq 0 ]]; then
-    echo "Successfully updated user in Fleet"
-else
-    echo "Failed to update user in Fleet"
-    echo $resp
-    exit 2
-fi

salt/common/tools/sbin/so-fleet-user-update

@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Copyright 2014-2022 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name>"
+    echo ""
+    echo "Update password for an existing Fleet user. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+MYSQL_PASS=$(lookup_pillar_secret mysql)
+FLEET_IP=$(lookup_pillar fleet_ip)
+FLEET_USER=$USER
+
+# test existence of user
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "SELECT count(1) FROM users WHERE email='$FLEET_USER'" 2>/dev/null | tail -1)
+if [[ $? -ne 0 ]] || [[ $MYSQL_OUTPUT -ne 1 ]] ; then
+    echo "Test for email [${FLEET_USER}] failed"
+    echo "    expect 1 hit in users database, return $MYSQL_OUTPUT hit(s)."
+    echo "Unable to update Fleet user password."
+    exit 2
+fi
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -rs FLEET_PASS
+
+if ! check_password "$FLEET_PASS"; then
+  echo "Password is invalid. Please exclude single quotes, double quotes, dollar signs, and backslashes from the password."
+  exit 2
+fi
+
+FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_PASS'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
+if [[ $? -ne 0 ]]; then
+	echo "Failed to generate Fleet password hash"
+	exit 2
+fi
+
+
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "UPDATE users SET password='$FLEET_HASH', salt='' where email='$FLEET_USER'" 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully updated Fleet user password"
+else
+    echo "Unable to update Fleet user password"
+    echo "$MYSQL_OUTPUT"
+    exit 2
+fi

salt/common/tools/sbin/so-grafana-dashboard-folder-delete

@@ -0,0 +1,17 @@
+# this script is used to delete the default Grafana dashboard folders that existed prior to Grafana dashboard and Salt management changes in 2.3.70
+
+folders=$(curl -X GET http://admin:{{salt['pillar.get']('secrets:grafana_admin')}}@localhost:3000/api/folders | jq -r '.[] | @base64')
+delfolder=("Manager" "Manager Search" "Sensor Nodes" "Search Nodes" "Standalone" "Eval Mode")
+
+for row in $folders; do
+   title=$(echo ${row} | base64 --decode | jq -r '.title')
+   uid=$(echo ${row} | base64 --decode | jq -r '.uid')
+
+  if [[ " ${delfolder[@]} " =~ " ${title} " ]]; then
+   curl -X DELETE http://admin:{{salt['pillar.get']('secrets:grafana_admin')}}@localhost:3000/api/folders/$uid
+  fi
+done
+
+echo "so-grafana-dashboard-folder-delete has been run to delete default Grafana dashboard folders that existed prior to 2.3.70" > /opt/so/state/so-grafana-dashboard-folder-delete-complete
+
+exit 0

salt/common/tools/sbin/so-grafana-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-grafana-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-grafana-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-idstools-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-idstools-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-idstools-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-image-common

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,6 +17,7 @@
 
 # NOTE: This script depends on so-common
 IMAGEREPO=security-onion-solutions
+STATUS_CONF='/opt/so/conf/so-status/so-status.conf'
 
 # shellcheck disable=SC2120
 container_list() {
@@ -131,12 +132,17 @@ update_docker_containers() {
   # Let's make sure we have the public key
   run_check_net_err \
   "curl --retry 5 --retry-delay 60 -sSL https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS -o $SIGNPATH/KEYS" \
-  "Could not pull signature key file, please ensure connectivity to https://raw.gihubusercontent.com" \
+  "Could not pull signature key file, please ensure connectivity to https://raw.githubusercontent.com" \
   noretry >> "$LOG_FILE" 2>&1
   result=$?
   if [[ $result -eq 0 ]]; then
     cat $SIGNPATH/KEYS | gpg --import - >> "$LOG_FILE" 2>&1
   fi
+
+  # If downloading for soup, check if any optional images need to be pulled
+  if [[ $CURLTYPE == 'soup' ]]; then
+    grep -q "so-logscan" "$STATUS_CONF" && TRUSTED_CONTAINERS+=("so-logscan")
+  fi
   
   # Download the containers from the interwebs
   for i in "${TRUSTED_CONTAINERS[@]}"

salt/common/tools/sbin/so-image-pull

@@ -0,0 +1,58 @@
+#!/bin/bash
+#
+# Copyright 2014-2022 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+. /usr/sbin/so-image-common
+
+usage() {
+	read -r -d '' message <<- EOM
+		usage: so-image-pull [-h] IMAGE [IMAGE ...]
+
+		positional arguments:
+			IMAGE         One or more 'so-' prefixed images to download and verify.
+
+		optional arguments:
+			-h, --help    Show this help message and exit.
+	EOM
+	echo "$message"
+	exit 1
+}
+
+for arg; do
+	shift
+	[[ "$arg" = "--quiet" || "$arg" = "-q" ]] && quiet=true && continue
+	set -- "$@" "$arg"
+done
+
+if [[ $# -eq 0 || $# -gt 1 ]] || [[ $1 == '-h' || $1 == '--help' ]]; then
+	usage
+fi
+
+TRUSTED_CONTAINERS=("$@")
+set_version
+
+for image in "${TRUSTED_CONTAINERS[@]}"; do
+	if ! docker images | grep "$image" | grep ":5000" | grep -q "$VERSION"; then
+		if [[ $quiet == true ]]; then
+			update_docker_containers "$image" "" "" "/dev/null"
+		else
+			update_docker_containers "$image" "" "" "" 
+		fi
+	else
+		echo "$image:$VERSION image exists."
+	fi
+done

salt/common/tools/sbin/so-import-evtx

@@ -0,0 +1,176 @@
+#!/bin/bash
+#
+# Copyright 2014-2022 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+{%- set MANAGER = salt['grains.get']('master') %}
+{%- set VERSION = salt['pillar.get']('global:soversion') %}
+{%- set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
+{%- set MANAGERIP = salt['pillar.get']('global:managerip') -%}
+{%- set URLBASE = salt['pillar.get']('global:url_base') %}
+{% set ES_USER = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:user', '') %}
+{% set ES_PASS = salt['pillar.get']('elasticsearch:auth:users:so_elastic_user:pass', '') %}
+
+INDEX_DATE=$(date +'%Y.%m.%d')
+RUNID=$(cat /dev/urandom | tr -dc 'a-z0-9' | fold -w 8 | head -n 1)
+LOG_FILE=/nsm/import/evtx-import.log
+
+. /usr/sbin/so-common
+
+function usage {
+  cat << EOF
+Usage: $0 <evtx-file-1> [evtx-file-2] [evtx-file-*]
+
+Imports one or more evtx files into Security Onion. The evtx files will be analyzed and made available for review in the Security Onion toolset.
+EOF
+}
+
+
+function evtx2es() {
+  EVTX=$1
+  HASH=$2
+  
+  ES_PASS=$(lookup_pillar "auth:users:so_elastic_user:pass" "elasticsearch")
+  ES_USER=$(lookup_pillar "auth:users:so_elastic_user:user" "elasticsearch")
+
+  docker run --rm \
+    -v "$EVTX:/tmp/$RUNID.evtx" \
+    --entrypoint evtx2es \
+    {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} \
+    --host {{ MANAGERIP }} --scheme https \
+    --index so-beats-$INDEX_DATE --pipeline import.wel \
+    --login $ES_USER --pwd "$ES_PASS" \
+    "/tmp/$RUNID.evtx" >> $LOG_FILE 2>&1
+
+  docker run --rm \
+    -v "$EVTX:/tmp/import.evtx" \
+    -v "/nsm/import/evtx-end_newest:/tmp/newest" \
+    -v "/nsm/import/evtx-start_oldest:/tmp/oldest" \
+    --entrypoint '/evtx_calc_timestamps.sh' \
+    {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }}
+}
+
+# if no parameters supplied, display usage
+if [ $# -eq 0 ]; then
+  usage
+  exit 1
+fi
+
+# ensure this is a Manager node
+require_manager
+
+# verify that all parameters are files
+for i in "$@"; do
+  if ! [ -f "$i" ]; then
+    usage
+    echo "\"$i\" is not a valid file!"
+    exit 2
+  fi
+done
+
+# track if we have any valid or invalid evtx
+INVALID_EVTXS="no"
+VALID_EVTXS="no"
+
+# track oldest start and newest end so that we can generate the Kibana search hyperlink at the end
+START_OLDEST="2050-12-31"
+END_NEWEST="1971-01-01"
+
+touch /nsm/import/evtx-start_oldest
+touch /nsm/import/evtx-end_newest
+
+echo $START_OLDEST > /nsm/import/evtx-start_oldest
+echo $END_NEWEST > /nsm/import/evtx-end_newest
+
+# paths must be quoted in case they include spaces
+for EVTX in "$@"; do
+  EVTX=$(/usr/bin/realpath "$EVTX")
+  echo "Processing Import: ${EVTX}"
+
+  # generate a unique hash to assist with dedupe checks
+  HASH=$(md5sum "${EVTX}" | awk '{ print $1 }')
+  HASH_DIR=/nsm/import/${HASH}
+  echo "- assigning unique identifier to import: $HASH"
+
+  if [ -d $HASH_DIR ]; then
+    echo "- this EVTX has already been imported; skipping"
+    INVALID_EVTXS="yes"
+  else
+    VALID_EVTXS="yes"
+
+    EVTX_DIR=$HASH_DIR/evtx
+    mkdir -p $EVTX_DIR
+
+    # import evtx and write them to import ingest pipeline
+    echo "- importing logs to Elasticsearch..."
+    evtx2es "${EVTX}" $HASH
+
+    # compare $START to $START_OLDEST
+    START=$(cat /nsm/import/evtx-start_oldest)
+    START_COMPARE=$(date -d $START +%s)
+    START_OLDEST_COMPARE=$(date -d $START_OLDEST +%s)
+    if [ $START_COMPARE -lt $START_OLDEST_COMPARE ]; then
+      START_OLDEST=$START
+    fi  
+
+    # compare $ENDNEXT to $END_NEWEST
+    END=$(cat /nsm/import/evtx-end_newest)
+    ENDNEXT=`date +%Y-%m-%d --date="$END 1 day"`
+    ENDNEXT_COMPARE=$(date -d $ENDNEXT +%s)
+    END_NEWEST_COMPARE=$(date -d $END_NEWEST +%s)
+    if [ $ENDNEXT_COMPARE -gt $END_NEWEST_COMPARE ]; then
+      END_NEWEST=$ENDNEXT
+    fi  
+
+    cp -f "${EVTX}" "${EVTX_DIR}"/data.evtx
+    chmod 644 "${EVTX_DIR}"/data.evtx
+
+  fi # end of valid evtx
+
+  echo
+
+done # end of for-loop processing evtx files
+
+# remove temp files
+echo "Cleaning up:"
+for TEMP_EVTX in ${TEMP_EVTXS[@]}; do
+  echo "- removing temporary evtx $TEMP_EVTX"
+  rm -f $TEMP_EVTX
+done
+
+# output final messages
+if [ "$INVALID_EVTXS" = "yes" ]; then
+  echo
+  echo "Please note!  One or more evtx was invalid!  You can scroll up to see which ones were invalid."
+fi
+
+START_OLDEST_FORMATTED=`date +%Y-%m-%d --date="$START_OLDEST"`
+START_OLDEST_SLASH=$(echo $START_OLDEST_FORMATTED | sed -e 's/-/%2F/g')
+END_NEWEST_SLASH=$(echo $END_NEWEST | sed -e 's/-/%2F/g')
+
+if [ "$VALID_EVTXS" = "yes" ]; then
+cat << EOF
+
+Import complete!
+
+You can use the following hyperlink to view data in the time range of your import.  You can triple-click to quickly highlight the entire hyperlink and you can then copy it into your browser:
+https://{{ URLBASE }}/#/hunt?q=import.id:${RUNID}%20%7C%20groupby%20event.module%20event.dataset&t=${START_OLDEST_SLASH}%2000%3A00%3A00%20AM%20-%20${END_NEWEST_SLASH}%2000%3A00%3A00%20AM&z=UTC
+
+or you can manually set your Time Range to be (in UTC):
+From: $START_OLDEST_FORMATTED    To: $END_NEWEST
+
+Please note that it may take 30 seconds or more for events to appear in Hunt.
+EOF
+fi

salt/common/tools/sbin/so-import-pcap

@@ -1,6 +1,6 @@
 #!/bin/bash
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-index-list

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-influxdb-clean

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-influxdb-downsample

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-influxdb-drop-autogen

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-influxdb-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-influxdb-start

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-influxdb-stop

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-ip-update

@@ -8,9 +8,9 @@ fi
 
 echo "This tool will update a manager's IP address to the new IP assigned to the management network interface."
 
-echo
+echo ""
 echo "WARNING: This tool is still undergoing testing, use at your own risk!"
-echo
+echo ""
 
 if [ -z "$OLD_IP" ]; then
 	OLD_IP=$(lookup_pillar "managerip")
@@ -27,7 +27,7 @@ if [ -z "$NEW_IP" ]; then
 	NEW_IP=$(ip -4 addr list $iface | grep inet | cut -d' ' -f6 | cut -d/ -f1)
 
 	if [ -z "$NEW_IP" ]; then
-		fail "Unable to detect new IP on interface $iface.    "
+		fail "Unable to detect new IP on interface $iface."
 	fi
 
 	echo "Detected new IP $NEW_IP on interface $iface."
@@ -39,15 +39,20 @@ fi
 
 echo "About to change old IP $OLD_IP to new IP $NEW_IP."
 
-echo
+echo ""
 read -n 1 -p "Would you like to continue? (y/N) " CONTINUE
-echo
+echo ""
 
 if [ "$CONTINUE" == "y" ]; then
-  for file in $(grep -rlI $OLD_IP /opt/so/saltstack /etc); do
-  	echo "Updating file: $file"
-    sed -i "s|$OLD_IP|$NEW_IP|g" $file
-  done
+	for file in $(grep -rlI $OLD_IP /opt/so/saltstack /etc); do
+		echo "Updating file: $file"
+		sed -i "s|$OLD_IP|$NEW_IP|g" $file
+	done
+
+	echo "Granting MySQL root user permissions on $NEW_IP"
+	docker exec -i so-mysql mysql --user=root --password=$(lookup_pillar_secret 'mysql') -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'$NEW_IP' IDENTIFIED BY '$(lookup_pillar_secret 'mysql')' WITH GRANT OPTION;" &> /dev/null
+	echo "Removing MySQL root user from $OLD_IP"
+	docker exec -i so-mysql mysql --user=root --password=$(lookup_pillar_secret 'mysql') -e "DROP USER 'root'@'$OLD_IP';" &> /dev/null
 
 	echo "The IP has been changed from $OLD_IP to $NEW_IP."
 
@@ -60,4 +65,4 @@ if [ "$CONTINUE" == "y" ]; then
 	fi
 else
 	echo "Exiting without changes."
-fi
+fi

salt/common/tools/sbin/so-kibana-config-export

@@ -5,7 +5,7 @@
 # {%- set FLEET_IP = salt['pillar.get']('global:fleet_ip', '') %}
 # {%- set MANAGER = salt['pillar.get']('global:url_base', '') %}
 #
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

salt/common/tools/sbin/so-kibana-restart

@@ -1,6 +1,6 @@
 #!/bin/bash
 
-# Copyright 2014,2015,2016,2017,2018,2019,2020,2021 Security Onion Solutions, LLC
+# Copyright 2014-2022 Security Onion Solutions, LLC
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by