Merge pull request #1363 from Security-Onion-Solutions/dev

RC3

README.md

@@ -1,6 +1,6 @@
-## Security Onion 2.0.1.rc1
+## Security Onion 2.2.0.rc3
 
-Security Onion 2.0.1 RC1 is here! This version requires a fresh install, but there is good news - we have brought back soup! From now on, you should be able to run soup on the manager to upgrade your environment to RC2 and beyond! 
+Security Onion 2.2.0 RC3 is here!
 
 ### Warnings and Disclaimers
 
@@ -14,24 +14,24 @@ Security Onion 2.0.1 RC1 is here! This version requires a fresh install, but the
 
 ### Release Notes
 
-https://docs.securityonion.net/en/2.0/release-notes.html
+https://docs.securityonion.net/en/2.2/release-notes.html
 
 ### Requirements
 
-https://docs.securityonion.net/en/2.0/hardware.html
+https://docs.securityonion.net/en/2.2/hardware.html
 
 ### Download
 
-https://docs.securityonion.net/en/2.0/download.html
+https://docs.securityonion.net/en/2.2/download.html
 
 ### Installation
 
-https://docs.securityonion.net/en/2.0/installation.html
+https://docs.securityonion.net/en/2.2/installation.html
 
 ### FAQ
 
-https://docs.securityonion.net/en/2.0/faq.html
+https://docs.securityonion.net/en/2.2/faq.html
 
 ### Feedback
 
-https://docs.securityonion.net/en/2.0/community-support.html
+https://docs.securityonion.net/en/2.2/community-support.html

VERIFY_ISO.md

@@ -1,16 +1,16 @@
-### 2.0.0-rc1 ISO image built on 2020/07/20
+### 2.2.0-rc3 ISO image built on 2020/09/17
 
 ### Download and Verify
 
-2.0.0-rc1 ISO image:  
-https://download.securityonion.net/file/securityonion/securityonion-2.0.0-rc1.iso
+2.2.0-rc3 ISO image:  
+https://download.securityonion.net/file/securityonion/securityonion-2.2.0-rc3.iso
 
-MD5: 788570E839439C23956581C6145B8689  
-SHA1: A87CAF016C989D4DB4D4ED619DF072B708BA28FE  
-SHA256: C5AC6419AF40CB98E93C53CE4101E7DE5F51AEE76DB46734191D783503649210  
+MD5: 051883501C905653ACBCEC513C294778  
+SHA1: 0A66F6636F53B268E7FFB743A3136AC5CC3E0E96  
+SHA256: 5A9F303954AF1B1D271CE526E5DCBFC28F3FFC0621B291A29F0F7F2E8EB11C43 
 
 Signature for ISO image:  
-https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.0.0-rc1.iso.sig
+https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.2.0-rc3.iso.sig
 
 Signing key:  
 https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/master/KEYS  
@@ -24,22 +24,22 @@ wget https://raw.githubusercontent.com/Security-Onion-Solutions/securityonion/ma
 
 Download the signature file for the ISO:  
 ```
-wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.0.0-rc1.iso.sig
+wget https://github.com/Security-Onion-Solutions/securityonion/raw/master/sigs/securityonion-2.2.0-rc3.iso.sig
 ```
 
 Download the ISO image:  
 ```
-wget https://download.securityonion.net/file/securityonion/securityonion-2.0.0-rc1.iso
+wget https://download.securityonion.net/file/securityonion/securityonion-2.2.0-rc3.iso
 ```
 
 Verify the downloaded ISO image using the signature file:  
 ```
-gpg --verify securityonion-2.0.0-rc1.iso.sig securityonion-2.0.0-rc1.iso
+gpg --verify securityonion-2.2.0-rc3.iso.sig securityonion-2.2.0-rc3.iso
 ```
 
 The output should show "Good signature" and the Primary key fingerprint should match what's shown below:
 ```
-gpg: Signature made Mon 20 Jul 2020 03:01:19 PM EDT using RSA key ID FE507013
+gpg: Signature made Thu 17 Sep 2020 10:05:27 AM EDT using RSA key ID FE507013
 gpg: Good signature from "Security Onion Solutions, LLC <info@securityonionsolutions.com>"
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
@@ -47,4 +47,4 @@ Primary key fingerprint: C804 A93D 36BE 0C73 3EA1  9644 7C10 60B7 FE50 7013
 ```
 
 Once you've verified the ISO image, you're ready to proceed to our Installation guide:  
-https://docs.securityonion.net/en/2.0/installation.html
+https://docs.securityonion.net/en/2.2/installation.html

VERSION

@@ -1 +1 @@
-2.0.1-rc.1
+2.2.0-rc.3

files/analyst/README

@@ -0,0 +1,79 @@
+The following GUI tools are available on the analyst workstation:
+
+chromium
+  url: https://www.chromium.org/Home
+  To run chromium, click Applications > Internet > Chromium Web Browser
+  
+Wireshark
+  url: https://www.wireshark.org/
+  To run Wireshark, click Applications > Internet > Wireshark Network Analyzer
+
+NetworkMiner
+  url: https://www.netresec.com
+  To run NetworkMiner, click Applications > Internet > NetworkMiner
+
+The following CLI tools are available on the analyst workstation:
+
+bit-twist
+  url: http://bittwist.sourceforge.net
+  To run bit-twist, open a terminal and type: bittwist -h
+
+chaosreader
+  url: http://chaosreader.sourceforge.net
+  To run chaosreader, open a terminal and type: chaosreader -h
+
+dnsiff
+  url: https://www.monkey.org/~dugsong/dsniff/
+  To run dsniff, open a terminal and type: dsniff -h
+
+foremost
+  url: http://foremost.sourceforge.net
+  To run foremost, open a terminal and type: foremost -h
+  
+hping3
+  url: http://www.hping.org/hping3.html
+  To run hping3, open a terminal and type: hping3 -h
+
+netsed
+  url: http://silicone.homelinux.org/projects/netsed/
+  To run netsed, open a terminal and type: netsed -h
+
+ngrep
+  url: https://github.com/jpr5/ngrep
+  To run ngrep, open a terminal and type: ngrep -h
+
+scapy
+  url: http://www.secdev.org/projects/scapy/
+  To run scapy, open a terminal and type: scapy
+
+ssldump
+  url: http://www.rtfm.com/ssldump/
+  To run ssldump, open a terminal and type: ssldump -h
+
+sslsplit
+  url: https://github.com/droe/sslsplit
+  To run sslsplit, open a terminal and type: sslsplit -h
+
+tcpdump
+  url: http://www.tcpdump.org
+  To run tcpdump, open a terminal and type: tcpdump -h
+
+tcpflow
+  url: https://github.com/simsong/tcpflow
+  To run tcpflow, open a terminal and type: tcpflow -h
+
+tcpstat
+  url: https://frenchfries.net/paul/tcpstat/
+  To run tcpstat, open a terminal and type: tcpstat -h
+
+tcptrace
+  url: http://www.tcptrace.org
+  To run tcptrace, open a terminal and type: tcptrace -h
+
+tcpxtract
+  url: http://tcpxtract.sourceforge.net/
+  To run tcpxtract, open a terminal and type: tcpxtract -h
+
+whois
+  url: http://www.linux.it/~md/software/
+  To run whois, open a terminal and type: whois -h

files/firewall/assigned_hostgroups.local.map.yaml

@@ -13,6 +13,7 @@ role:
   fleet:
   heavynode:
   helixsensor:
+  import:
   manager:
   managersearch:
   standalone:

pillar/data/addtotab.sh

@@ -44,11 +44,11 @@ echo "    guid: $GUID" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    rootfs: $ROOTFS" >> $local_salt_dir/pillar/data/$TYPE.sls
 echo "    nsmfs: $NSM" >> $local_salt_dir/pillar/data/$TYPE.sls
 if [ $TYPE == 'sensorstab' ]; then
-  echo "    monint: $MONINT" >> $local_salt_dir/pillar/data/$TYPE.sls
+  echo "    monint: bond0" >> $local_salt_dir/pillar/data/$TYPE.sls
   salt-call state.apply grafana queue=True
 fi
 if [ $TYPE == 'evaltab' ] || [ $TYPE == 'standalonetab' ]; then
-  echo "    monint: $MONINT" >> $local_salt_dir/pillar/data/$TYPE.sls
+  echo "    monint: bond0" >> $local_salt_dir/pillar/data/$TYPE.sls
   if [ ! $10 ]; then
     salt-call state.apply grafana queue=True
     salt-call state.apply utility queue=True

pillar/docker/config.sls

@@ -1,11 +1,11 @@
-{%- set FLEETMANAGER = salt['pillar.get']('static:fleet_manager', False) -%}
-{%- set FLEETNODE = salt['pillar.get']('static:fleet_node', False) -%}
+{%- set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
+{%- set FLEETNODE = salt['pillar.get']('global:fleet_node', False) -%}
 {% set WAZUH = salt['pillar.get']('manager:wazuh', '0') %}
 {% set THEHIVE = salt['pillar.get']('manager:thehive', '0') %}
 {% set PLAYBOOK = salt['pillar.get']('manager:playbook', '0') %}
 {% set FREQSERVER = salt['pillar.get']('manager:freq', '0') %}
 {% set DOMAINSTATS = salt['pillar.get']('manager:domainstats', '0') %}
-{% set ZEEKVER = salt['pillar.get']('static:zeekversion', 'COMMUNITY') %}
+{% set ZEEKVER = salt['pillar.get']('global:mdengine', 'COMMUNITY') %}
 {% set GRAFANA = salt['pillar.get']('manager:grafana', '0') %}
 
 eval:

pillar/firewall/ports.sls

@@ -26,6 +26,7 @@ firewall:
         - 4200
         - 5601
         - 6379
+        - 7788
         - 8086
         - 8090
         - 9001
@@ -33,6 +34,8 @@ firewall:
         - 9300
         - 9400  
         - 9500
+        - 9595
+        - 9696
       udp:
         - 1514
   minions:

pillar/logstash/init.sls

@@ -1,7 +1,6 @@
 logstash:
   docker_options:
     port_bindings:
-      - 0.0.0.0:514:514
       - 0.0.0.0:5044:5044
       - 0.0.0.0:5644:5644
       - 0.0.0.0:6050:6050

pillar/logstash/manager.sls

@@ -1,3 +1,4 @@
+{%- set PIPELINE = salt['pillar.get']('global:pipeline', 'redis') %}
 logstash:
   pipelines:
     manager:
@@ -5,3 +6,4 @@ logstash:
         - so/0009_input_beats.conf      
         - so/0010_input_hhbeats.conf
         - so/9999_output_redis.conf.jinja
+        

pillar/logstash/search.sls

@@ -1,3 +1,4 @@
+{%- set PIPELINE = salt['pillar.get']('global:pipeline', 'minio') %}
 logstash:
   pipelines:
     search:

pillar/top.sls

@@ -2,7 +2,7 @@ base:
   '*':
     - patch.needs_restarting
 
-  '*_eval or *_helix or *_heavynode or *_sensor or *_standalone':
+  '*_eval or *_helix or *_heavynode or *_sensor or *_standalone or *_import':
     - match: compound
     - zeek
 
@@ -14,14 +14,14 @@ base:
     - elasticsearch.search
 
   '*_sensor':
-    - static
+    - global
     - zeeklogs
     - healthcheck.sensor
     - minions.{{ grains.id }}
 
   '*_manager or *_managersearch':
     - match: compound
-    - static
+    - global
     - data.*
     - secrets
     - minions.{{ grains.id }}
@@ -36,7 +36,7 @@ base:
     - secrets
     - healthcheck.eval
     - elasticsearch.eval
-    - static
+    - global
     - minions.{{ grains.id }}
 
   '*_standalone':
@@ -48,20 +48,20 @@ base:
     - zeeklogs
     - secrets
     - healthcheck.standalone
-    - static
+    - global
     - minions.{{ grains.id }}
 
   '*_node':
-    - static
+    - global
     - minions.{{ grains.id }}
 
   '*_heavynode':
-    - static
+    - global
     - zeeklogs
     - minions.{{ grains.id }}
 
   '*_helix':
-    - static
+    - global
     - fireeye
     - zeeklogs
     - logstash
@@ -69,14 +69,21 @@ base:
     - minions.{{ grains.id }}
 
   '*_fleet':
-    - static
+    - global
     - data.*
     - secrets
     - minions.{{ grains.id }}
 
   '*_searchnode':
-    - static
+    - global
     - logstash
     - logstash.search
     - elasticsearch.search
     - minions.{{ grains.id }}
+
+  '*_import':
+    - zeeklogs
+    - secrets
+    - elasticsearch.eval
+    - global
+    - minions.{{ grains.id }}

salt/_modules/healthcheck.py

@@ -2,6 +2,8 @@
 
 import logging
 import sys
+from time import time
+from os.path import getsize
 
 allowed_functions = ['is_enabled', 'zeek']
 states_to_apply = []
@@ -85,8 +87,21 @@ def zeek():
   else:
     zeek_restart = 0
 
-  __salt__['telegraf.send']('healthcheck zeek_restart=%i' % zeek_restart)
-  
+  #__salt__['telegraf.send']('healthcheck zeek_restart=%i' % zeek_restart)
+  # write out to file in /nsm/zeek/logs/ for telegraf to read for zeek restart
+  try:
+    if getsize("/nsm/zeek/logs/zeek_restart.log") >= 1000000:
+      openmethod = "w"
+    else:
+      openmethod = "a"
+  except FileNotFoundError:
+    openmethod = "a"
+
+  influxtime = int(time() * 1000000000)
+  with open("/nsm/zeek/logs/zeek_restart.log", openmethod) as f:
+    f.write('healthcheck zeek_restart=%i %i\n' % (zeek_restart, influxtime))
+
+
   if calling_func == 'execute' and zeek_restart:
     apply_states()
   

salt/_modules/so.py

@@ -0,0 +1,4 @@
+#!py
+
+def status():
+  return __salt__['cmd.run']('/sbin/so-status')

salt/airgap/files/yum.conf

@@ -0,0 +1,12 @@
+[main]
+cachedir=/var/cache/yum/$basearch/$releasever
+keepcache=0
+debuglevel=2
+logfile=/var/log/yum.log
+exactarch=1
+obsoletes=1
+gpgcheck=1
+plugins=1
+installonly_limit=2
+bugtracker_url=http://bugs.centos.org/set_project.php?project_id=23&ref=http://bugs.centos.org/bug_report_page.php?category=yum
+distroverpkg=centos-release

salt/airgap/init.sls

@@ -0,0 +1,60 @@
+{% set MANAGER = salt['grains.get']('master') %}
+airgapyum:
+  file.managed:
+    - name: /etc/yum/yum.conf
+    - source: salt://airgap/files/yum.conf
+
+airgap_repo:
+  pkgrepo.managed:
+    - humanname: Airgap Repo
+    - baseurl: https://{{ MANAGER }}/repo
+    - gpgcheck: 0
+    - sslverify: 0
+
+agbase:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-Base.repo
+
+agcr:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-CR.repo
+
+agdebug:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-Debuginfo.repo
+
+agfasttrack:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-fasttrack.repo
+
+agmedia:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-Media.repo
+
+agsources:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-Sources.repo
+
+agvault:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-Vault.repo
+
+agkernel:
+  file.absent:
+    - name: /etc/yum.repos.d/CentOS-x86_64-kernel.repo
+
+agepel:
+  file.absent:
+    - name: /etc/yum.repos.d/epel.repo
+
+agtesting:
+  file.absent:
+    - name: /etc/yum.repos.d/epel-testing.repo
+
+agssrepo:
+  file.absent:
+    - name: /etc/yum.repos.d/saltstack.repo
+
+agwazrepo:
+  file.absent:
+    - name: /etc/yum.repos.d/wazuh.repo

salt/ca/init.sls

@@ -1,3 +1,8 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'ca' in top_states %}
+
 {% set manager = salt['grains.get']('master') %}
 /etc/salt/minion.d/signing_policies.conf:
   file.managed:
@@ -10,12 +15,16 @@
   file.directory: []
 
 pki_private_key:
-    x509.private_key_managed:
-        - name: /etc/pki/ca.key
-        - bits: 4096
-        - passphrase:
-        - cipher: aes_256_cbc
-        - backup: True
+  x509.private_key_managed:
+    - name: /etc/pki/ca.key
+    - bits: 4096
+    - passphrase:
+    - cipher: aes_256_cbc
+    - backup: True
+    {% if salt['file.file_exists']('/etc/pki/ca.key') -%}
+    - prereq:
+      - x509: /etc/pki/ca.crt
+    {%- endif %}
 
 /etc/pki/ca.crt:
   x509.certificate_managed:
@@ -32,18 +41,15 @@ pki_private_key:
     - days_valid: 3650
     - days_remaining: 0
     - backup: True
-    - managed_private_key:
-        name: /etc/pki/ca.key
-        bits: 4096
-        backup: True
+    - replace: False
     - require:
       - file: /etc/pki
 
-send_x509_pem_entries_to_mine:
+x509_pem_entries:
   module.run:
     - mine.send:
-      - func: x509.get_pem_entries
-      - glob_path: /etc/pki/ca.crt
+       - name: x509.get_pem_entries
+       - glob_path: /etc/pki/ca.crt
 
 cakeyperms:
   file.managed:
@@ -51,3 +57,11 @@ cakeyperms:
     - name: /etc/pki/ca.key
     - mode: 640
     - group: 939
+
+{% else %}
+
+ca_state_not_allowed:
+  test.fail_without_changes:
+    - name: ca_state_not_allowed
+
+{% endif %}

salt/common/cron/sensor-rotate

@@ -0,0 +1,2 @@
+#!/bin/bash
+/usr/sbin/logrotate -f /opt/so/conf/sensor-rotate.conf > /dev/null 2>&1

salt/common/files/sensor-rotate.conf

@@ -0,0 +1,10 @@
+/opt/so/log/sensor_clean.log
+{
+    daily
+    rotate 2
+    missingok
+    nocompress
+    create
+    sharedscripts
+    endscript
+}

salt/common/init.sls

@@ -1,3 +1,8 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'common' in top_states %}
+
 {% set role = grains.id.split('_') | last %}
 
 # Remove variables.txt from /tmp - This is temp
@@ -88,7 +93,7 @@ heldpackages:
   pkg.installed:
     - pkgs:
       - containerd.io: 1.2.13-2
-      - docker-ce: 5:19.03.9~3-0~ubuntu-bionic
+      - docker-ce: 5:19.03.12~3-0~ubuntu-bionic
     - hold: True
     - update_holds: True
 
@@ -124,7 +129,7 @@ heldpackages:
   pkg.installed:
     - pkgs:
       - containerd.io: 1.2.13-3.2.el7
-      - docker-ce: 3:19.03.11-3.el7
+      - docker-ce: 3:19.03.12-3.el7
     - hold: True
     - update_holds: True
 {% endif %}
@@ -163,4 +168,39 @@ utilsyncscripts:
     - daymonth: '*'
     - month: '*'
     - dayweek: '*'
+
+sensorrotatescript:
+  file.managed:
+    - name: /usr/local/bin/sensor-rotate
+    - source: salt://common/cron/sensor-rotate
+    - mode: 755
+
+sensorrotateconf:
+  file.managed:
+    - name: /opt/so/conf/sensor-rotate.conf
+    - source: salt://common/files/sensor-rotate.conf
+    - mode: 644
+
+/usr/local/bin/sensor-rotate:
+  cron.present:
+    - user: root
+    - minute: '1'
+    - hour: '0'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
 {% endif %}
+
+# Make sure Docker is always running
+docker:
+  service.running:
+    - enable: True
+
+{% else %}
+
+common_state_not_allowed:
+  test.fail_without_changes:
+    - name: common_state_not_allowed
+
+{% endif %}

salt/common/maps/import.map.jinja

@@ -0,0 +1,10 @@
+{% set docker = {
+  'containers': [
+    'so-filebeat',
+    'so-nginx',
+    'so-soc',
+    'so-kratos',
+    'so-elasticsearch',
+    'so-kibana'
+  ]
+} %}

salt/common/maps/mdengine.map.jinja

@@ -2,4 +2,4 @@
   'containers': [
     'so-zeek'
   ]
-} %}
+} %}

salt/common/maps/so-status.map.jinja

@@ -5,6 +5,9 @@
 # to the list predefined by the role / minion id affix
 {% macro append_containers(pillar_name, k, compare )%}
   {% if salt['pillar.get'](pillar_name~':'~k, {}) != compare %}
+    {% if k == 'enabled' %}
+      {% set k = pillar_name %}
+    {% endif %}
     {% from 'common/maps/'~k~'.map.jinja' import docker as d with context %}
     {% for li in d['containers'] %}
       {{ docker['containers'].append(li) }}
@@ -20,8 +23,8 @@
 
 {% if role in ['eval', 'managersearch', 'manager', 'standalone'] %}
   {{ append_containers('manager', 'grafana', 0) }}
-  {{ append_containers('static', 'fleet_manager', 0) }}
-  {{ append_containers('manager', 'wazuh', 0) }}
+  {{ append_containers('global', 'fleet_manager', 0) }}
+  {{ append_containers('global', 'wazuh', 0) }}
   {{ append_containers('manager', 'thehive', 0) }}
   {{ append_containers('manager', 'playbook', 0) }}
   {{ append_containers('manager', 'freq', 0) }}
@@ -29,11 +32,11 @@
 {% endif %}
 
 {% if role in ['eval', 'heavynode', 'sensor', 'standalone'] %}
-  {{ append_containers('static', 'strelka', 0) }}
+  {{ append_containers('strelka', 'enabled', 0) }}
 {% endif %}
 
 {% if role in ['heavynode', 'standalone'] %}
-  {{ append_containers('static', 'zeekversion', 'SURICATA') }}
+  {{ append_containers('global', 'mdengine', 'SURICATA') }}
 {% endif %}
 
 {% if role == 'searchnode' %}
@@ -41,5 +44,5 @@
 {% endif %}
 
 {% if role == 'sensor' %}
-  {{ append_containers('static', 'zeekversion', 'SURICATA') }}
+  {{ append_containers('global', 'mdengine', 'SURICATA') }}
 {% endif %}

salt/common/tools/sbin/so-allow

@@ -21,6 +21,30 @@ local_salt_dir=/opt/so/saltstack/local
 
 SKIP=0
 
+function usage {
+
+cat << EOF
+
+Usage: $0 [-abefhoprsw] [ -i IP ]
+
+This program allows you to add a firewall rule to allow connections from a new IP address or CIDR range.
+
+If you run this program with no arguments, it will present a menu for you to choose your options.
+
+If you want to automate and skip the menu, you can pass the desired options as command line arguments.
+
+EXAMPLES
+
+To add 10.1.2.3 to the analyst role:
+so-allow -a -i 10.1.2.3
+
+To add 10.1.2.0/24 to the osquery role:
+so-allow -o -i 10.1.2.0/24
+
+EOF
+
+}
+
 while getopts "ahfesprbowi:" OPTION
 do
     case $OPTION in
@@ -36,7 +60,7 @@ do
             FULLROLE="beats_endpoint"
             SKIP=1
             ;;
-	    e)
+	e)
             FULLROLE="elasticsearch_rest"
             SKIP=1
             ;;
@@ -127,7 +151,7 @@ salt-call state.apply firewall queue=True
 if grep -q -R "wazuh: 1" $local_salt_dir/pillar/*; then
     # If analyst, add to Wazuh AR whitelist
     if [ "$FULLROLE" == "analyst" ]; then
-        WAZUH_MGR_CFG="/opt/so/wazuh/etc/ossec.conf"
+        WAZUH_MGR_CFG="/nsm/wazuh/etc/ossec.conf"
         if ! grep -q "<white_list>$IP</white_list>" $WAZUH_MGR_CFG ; then
             DATE=$(date)
             sed -i 's/<\/ossec_config>//' $WAZUH_MGR_CFG

salt/common/tools/sbin/so-common

@@ -19,14 +19,29 @@ IMAGEREPO=securityonion
 
 # Check for prerequisites
 if [ "$(id -u)" -ne 0 ]; then
-        echo "This script must be run using sudo!"
-        exit 1
+    echo "This script must be run using sudo!"
+    exit 1
 fi
 
 # Define a banner to separate sections
 banner="========================================================================="
 
 header() {
-        echo
-        printf '%s\n' "$banner" "$*" "$banner"
+	echo
+    printf '%s\n' "$banner" "$*" "$banner"
 }
+
+lookup_pillar() {
+    key=$1
+    cat /opt/so/saltstack/local/pillar/global.sls | grep $key | awk '{print $2}'
+}
+
+lookup_pillar_secret() {
+    key=$1
+    cat /opt/so/saltstack/local/pillar/secrets.sls | grep $key | awk '{print $2}'
+}
+
+check_container() {
+	docker ps | grep "$1:" > /dev/null 2>&1
+	return $?
+}

salt/common/tools/sbin/so-cortex-user-add

@@ -0,0 +1,54 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <new-user-name>"
+    echo ""
+    echo "Adds a new user to Cortex. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+CORTEX_KEY=$(lookup_pillar cortexkey)
+CORTEX_IP=$(lookup_pillar managerip)
+CORTEX_ORG_NAME=$(lookup_pillar cortexorgname)
+CORTEX_USER=$USER
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -s CORTEX_PASS
+
+# Create new user in Cortex
+resp=$(curl -sk -XPOST -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" "https://$CORTEX_IP/cortex/api/user" -d "{\"name\": \"$CORTEX_USER\",\"roles\": [\"read\",\"analyze\",\"orgadmin\"],\"organization\": \"$CORTEX_ORG_NAME\",\"login\": \"$CORTEX_USER\",\"password\" : \"$CORTEX_PASS\" }")
+if [[ "$resp" =~ \"status\":\"Ok\" ]]; then
+    echo "Successfully added user to Cortex."
+else
+    echo "Unable to add user to Cortex; user might already exist."
+    echo $resp
+    exit 2
+fi
+    

salt/common/tools/sbin/so-cortex-user-enable

@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name> <true|false>"
+    echo ""
+    echo "Enables or disables a user in Cortex."
+    exit 1
+}
+
+if [ $# -ne 2 ]; then
+  usage
+fi
+
+USER=$1
+
+CORTEX_KEY=$(lookup_pillar cortexkey)
+CORTEX_IP=$(lookup_pillar managerip)
+CORTEX_USER=$USER
+
+case "${2^^}" in
+	FALSE | NO | 0)
+		CORTEX_STATUS=Locked
+		;;
+	TRUE | YES | 1)
+		CORTEX_STATUS=Ok
+		;;
+	*)
+		usage
+		;;
+esac
+
+resp=$(curl -sk -XPATCH -H "Authorization: Bearer $CORTEX_KEY" -H "Content-Type: application/json" "https://$CORTEX_IP/cortex/api/user/${CORTEX_USER}" -d "{\"status\":\"${CORTEX_STATUS}\" }")
+if [[ "$resp" =~ \"status\":\"Locked\" || "$resp" =~ \"status\":\"Ok\" ]]; then
+    echo "Successfully updated user in Cortex."
+else
+    echo "Failed to update user in Cortex."
+    echo $resp
+    exit 2
+fi
+    

salt/common/tools/sbin/so-docker-refresh

@@ -76,6 +76,7 @@ if [ $MANAGERCHECK != 'so-helix' ]; then
     "so-kibana:$VERSION" \
     "so-kratos:$VERSION" \
     "so-logstash:$VERSION" \
+    "so-minio:$VERSION" \
     "so-mysql:$VERSION" \
     "so-nginx:$VERSION" \
     "so-pcaptools:$VERSION" \

salt/common/tools/sbin/so-elastic-clear

@@ -14,7 +14,7 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-{%- set MANAGERIP = salt['pillar.get']('static:managerip', '') -%}
+{%- set MANAGERIP = salt['pillar.get']('global:managerip', '') -%}
 . /usr/sbin/so-common
 
 SKIP=0

salt/common/tools/sbin/so-elasticsearch-templates

@@ -1,4 +1,6 @@
-{% set MANAGERIP = salt['pillar.get']('manager:mainip', '') %}
+{%- set mainint = salt['pillar.get']('host:mainint') %}
+{%- set MYIP = salt['grains.get']('ip_interfaces:' ~ mainint)[0] %}
+
 #!/bin/bash
 # Copyright 2014,2015,2016,2017,2018,2019 Security Onion Solutions, LLC
 #
@@ -16,7 +18,7 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 default_conf_dir=/opt/so/conf
-ELASTICSEARCH_HOST="{{ MANAGERIP}}"
+ELASTICSEARCH_HOST="{{ MYIP }}"
 ELASTICSEARCH_PORT=9200
 #ELASTICSEARCH_AUTH=""
 

salt/common/tools/sbin/so-features-enable

@@ -17,6 +17,28 @@
 . /usr/sbin/so-common
 local_salt_dir=/opt/so/saltstack/local
 
+cat << EOF
+This program will switch from the open source version of the Elastic Stack to the Features version licensed under the Elastic license.
+If you proceed, then we will download new Docker images and restart services.
+
+Please review the Elastic license:
+https://raw.githubusercontent.com/elastic/elasticsearch/master/licenses/ELASTIC-LICENSE.txt
+
+Please also note that, if you have a distributed deployment and continue with this change, Elastic traffic between nodes will change from encrypted to cleartext!
+(We expect to support Elastic Features Security at some point in the future.)
+
+Do you agree to the terms of the Elastic license and understand the note about encryption?
+
+If so, type AGREE to accept the Elastic license and continue.  Otherwise, just press Enter to exit this program without making any changes.
+EOF
+
+read INPUT
+if [ "$INPUT" != "AGREE" ]; then
+        exit
+fi
+
+echo "Please wait while switching to Elastic Features."
+
 manager_check() {
   # Check to see if this is a manager
   MANAGERCHECK=$(cat /etc/salt/grains | grep role | awk '{print $2}')
@@ -29,9 +51,9 @@ manager_check() {
 }
 
 manager_check
-VERSION=$(grep soversion $local_salt_dir/pillar/static.sls | cut -d':' -f2|sed 's/ //g')
-# Modify static.sls to enable Features
-sed -i 's/features: False/features: True/' $local_salt_dir/pillar/static.sls
+VERSION=$(grep soversion $local_salt_dir/pillar/global.sls | cut -d':' -f2|sed 's/ //g')
+# Modify global.sls to enable Features
+sed -i 's/features: False/features: True/' $local_salt_dir/pillar/global.sls
 SUFFIX="-features"
 TRUSTED_CONTAINERS=( \
   "so-elasticsearch:$VERSION$SUFFIX" \

salt/common/tools/sbin/so-fleet-user-add

@@ -0,0 +1,59 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <new-user-name>"
+    echo ""
+    echo "Adds a new user to Fleet. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+MYSQL_PASS=$(lookup_pillar_secret mysql)
+FLEET_IP=$(lookup_pillar fleet_ip)
+FLEET_USER=$USER
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -s FLEET_PASS
+
+FLEET_HASH=$(docker exec so-soctopus python -c "import bcrypt; print(bcrypt.hashpw('$FLEET_PASS'.encode('utf-8'), bcrypt.gensalt()).decode('utf-8'));" 2>&1)
+if [[ $? -ne 0 ]]; then
+	echo "Failed to generate Fleet password hash."
+	exit 2
+fi
+
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "INSERT INTO users (password,salt,username,email,admin,enabled) VALUES ('$FLEET_HASH','','$FLEET_USER','$FLEET_USER',1,1)" 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully added user to Fleet."
+else
+    echo "Unable to add user to Fleet; user might already exist."
+    echo $resp
+    exit 2
+fi

salt/common/tools/sbin/so-fleet-user-enable

@@ -0,0 +1,58 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name>"
+    echo ""
+    echo "Enables or disables a user in Fleet."
+    exit 1
+}
+
+if [ $# -ne 2 ]; then
+  usage
+fi
+
+USER=$1
+
+MYSQL_PASS=$(lookup_pillar_secret mysql)
+FLEET_IP=$(lookup_pillar fleet_ip)
+FLEET_USER=$USER
+
+case "${2^^}" in
+    FALSE | NO | 0)
+        FLEET_STATUS=0
+        ;;
+    TRUE | YES | 1)
+        FLEET_STATUS=1
+        ;;
+    *)
+        usage
+        ;;
+esac
+
+MYSQL_OUTPUT=$(docker exec so-mysql mysql -u root --password=$MYSQL_PASS fleet -e \
+    "UPDATE users SET enabled=$FLEET_STATUS WHERE username='$FLEET_USER'" 2>&1)
+
+if [[ $? -eq 0 ]]; then
+    echo "Successfully updated user in Fleet."
+else
+    echo "Failed to update user in Fleet."
+    echo $resp
+    exit 2
+fi

salt/common/tools/sbin/so-import-pcap

@@ -15,10 +15,13 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-{% set MANAGER = salt['grains.get']('master') %}
-{% set VERSION = salt['pillar.get']('static:soversion') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
-{%- set MANAGERIP = salt['pillar.get']('static:managerip') -%}
+{%- set MANAGER = salt['grains.get']('master') %}
+{%- set VERSION = salt['pillar.get']('global:soversion') %}
+{%- set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
+{%- set MANAGERIP = salt['pillar.get']('global:managerip') -%}
+{%- set URLBASE = salt['pillar.get']('global:url_base') %}
+
+. /usr/sbin/so-common
 
 function usage {
   cat << EOF
@@ -32,13 +35,13 @@ EOF
 function pcapinfo() {
   PCAP=$1
   ARGS=$2
-  docker run --rm -v $PCAP:/input.pcap --entrypoint capinfos {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} /input.pcap $ARGS
+  docker run --rm -v "$PCAP:/input.pcap" --entrypoint capinfos {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} /input.pcap $ARGS
 }
 
 function pcapfix() {
   PCAP=$1
   PCAP_OUT=$2
-  docker run --rm -v $PCAP:/input.pcap -v $PCAP_OUT:$PCAP_OUT --entrypoint pcapfix {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} /input.pcap -o $PCAP_OUT > /dev/null 2>&1
+  docker run --rm -v "$PCAP:/input.pcap" -v $PCAP_OUT:$PCAP_OUT --entrypoint pcapfix {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-pcaptools:{{ VERSION }} /input.pcap -o $PCAP_OUT > /dev/null 2>&1
 }
 
 function suricata() {
@@ -57,7 +60,7 @@ function suricata() {
     -v /opt/so/conf/suricata/rules:/etc/suricata/rules:ro \
     -v ${LOG_PATH}:/var/log/suricata/:rw \
     -v ${NSM_PATH}/:/nsm/:rw \
-    -v $PCAP:/input.pcap:ro \
+    -v "$PCAP:/input.pcap:ro" \
     -v /opt/so/conf/suricata/bpf:/etc/suricata/bpf:ro \
     {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-suricata:{{ VERSION }} \
     --runmode single -k none -r /input.pcap > $LOG_PATH/console.log 2>&1
@@ -76,7 +79,7 @@ function zeek() {
     -v $NSM_PATH/logs:/nsm/zeek/logs:rw \
     -v $NSM_PATH/spool:/nsm/zeek/spool:rw \
     -v $NSM_PATH/extracted:/nsm/zeek/extracted:rw \
-    -v $PCAP:/input.pcap:ro \
+    -v "$PCAP:/input.pcap:ro" \
     -v /opt/so/conf/zeek/local.zeek:/opt/zeek/share/zeek/site/local.zeek:ro \
     -v /opt/so/conf/zeek/node.cfg:/opt/zeek/etc/node.cfg:ro \
     -v /opt/so/conf/zeek/zeekctl.cfg:/opt/zeek/etc/zeekctl.cfg:ro \
@@ -210,9 +213,9 @@ cat << EOF
 Import complete!
 
 You can use the following hyperlink to view data in the time range of your import.  You can triple-click to quickly highlight the entire hyperlink and you can then copy it into your browser:
-https://{{ MANAGERIP }}/#/hunt?q=%2a%20%7C%20groupby%20event.module%20event.dataset&t=${START_OLDEST_SLASH}%2000%3A00%3A00%20AM%20-%20${END_NEWEST_SLASH}%2000%3A00%3A00%20AM
+https://{{ URLBASE }}/#/hunt?q=import.id:${HASH}%20%7C%20groupby%20event.module%20event.dataset&t=${START_OLDEST_SLASH}%2000%3A00%3A00%20AM%20-%20${END_NEWEST_SLASH}%2000%3A00%3A00%20AM&z=UTC
 
-or you can manually set your Time Range to be:
+or you can manually set your Time Range to be (in UTC):
 From: $START_OLDEST    To: $END_NEWEST
 
 Please note that it may take 30 seconds or more for events to appear in Onion Hunt.

salt/common/tools/sbin/so-kibana-config-export

@@ -1,9 +1,9 @@
 #!/bin/bash
 #
-# {%- set FLEET_MANAGER = salt['pillar.get']('static:fleet_manager', False) -%}
-# {%- set FLEET_NODE = salt['pillar.get']('static:fleet_node', False) -%}
-# {%- set FLEET_IP = salt['pillar.get']('static:fleet_ip', '') %}
-# {%- set MANAGER = salt['pillar.get']('manager:url_base', '') %}
+# {%- set FLEET_MANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
+# {%- set FLEET_NODE = salt['pillar.get']('global:fleet_node', False) -%}
+# {%- set FLEET_IP = salt['pillar.get']('global:fleet_ip', '') %}
+# {%- set MANAGER = salt['pillar.get']('global:url_base', '') %}
 #
 # Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
 #

salt/common/tools/sbin/so-playbook-sync

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-docker exec so-soctopus python3 playbook_play-sync.py >> /opt/so/log/soctopus/so-playbook-sync.log 2>&1
+docker exec so-soctopus python3 playbook_play-sync.py

salt/common/tools/sbin/so-restart

@@ -19,18 +19,22 @@
 
 . /usr/sbin/so-common
 
-echo $banner
-printf "Restarting $1...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
-echo $banner
+if [ $# -ge 1 ]; then
 
-if [ "$2" = "--force" ]
-then
-   printf "\nForce-stopping all Salt jobs before proceeding\n\n"
-   salt-call saltutil.kill_all_jobs
+        echo $banner
+        printf "Restarting $1...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
+        echo $banner
+
+        if [ "$2" = "--force" ]; then
+                printf "\nForce-stopping all Salt jobs before proceeding\n\n"
+                salt-call saltutil.kill_all_jobs
+        fi
+
+        case $1 in
+                "cortex") docker stop so-thehive-cortex so-thehive && docker rm so-thehive-cortex so-thehive && salt-call state.apply hive queue=True;;
+                "steno") docker stop so-steno && docker rm so-steno && salt-call state.apply pcap queue=True;;
+                *)  docker stop so-$1 ; docker rm so-$1 ; salt-call state.apply $1 queue=True;;
+        esac
+else
+        echo -e "\nPlease provide an argument by running like so-restart $component, or by using the component-specific script.\nEx. so-restart filebeat, or so-filebeat-restart\n"
 fi
-
-case $1 in
-   "cortex") docker stop so-thehive-cortex so-thehive && docker rm so-thehive-cortex so-thehive && salt-call state.apply hive queue=True;;
-   "steno") docker stop so-steno && docker rm so-steno && salt-call state.apply pcap queue=True;;
-   *)  docker stop so-$1 ; docker rm so-$1 ; salt-call state.apply $1 queue=True;;
-esac

salt/common/tools/sbin/so-rule-update

@@ -10,4 +10,4 @@ got_root() {
 }
 
 got_root
-docker exec -it so-idstools /bin/bash -c 'cd /opt/so/idstools/etc && idstools-rulecat'
+docker exec -d so-idstools /bin/bash -c 'cd /opt/so/idstools/etc && idstools-rulecat'

salt/common/tools/sbin/so-sensor-clean

@@ -115,7 +115,5 @@ if [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ]; then
          clean
          CUR_USAGE=$(df -P $SENSOR_DIR | tail -1 | awk '{print $5}' | tr -d %)
     done
-else
-    echo "$(date) - Current usage value of $CUR_USAGE not greater than CRIT_DISK_USAGE value of $CRIT_DISK_USAGE..." >> $LOG
 fi
 

salt/common/tools/sbin/so-start

@@ -19,18 +19,21 @@
 
 . /usr/sbin/so-common
 
-echo $banner
-printf "Starting $1...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
-echo $banner
+if [ $# -ge 1 ]; then
+	echo $banner
+	printf "Starting $1...\n\nThis could take a while if another Salt job is running. \nRun this command with --force to stop all Salt jobs before proceeding.\n"
+	echo $banner
 
-if [ "$2" = "--force" ]
-then
-   printf "\nForce-stopping all Salt jobs before proceeding\n\n"
-   salt-call saltutil.kill_all_jobs
+	if [ "$2" = "--force" ]; then
+   		printf "\nForce-stopping all Salt jobs before proceeding\n\n"
+   		salt-call saltutil.kill_all_jobs
+	fi
+
+	case $1 in
+   		"all") salt-call state.highstate queue=True;;
+   		"steno") if docker ps | grep -q so-$1; then printf "\n$1 is already running!\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply pcap queue=True; fi ;; 
+   		*) if docker ps | grep -E -q '^so-$1$'; then printf "\n$1 is already running\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply $1 queue=True; fi ;; 
+	esac
+else
+	echo -e "\nPlease provide an argument by running like so-start $component, or by using the component-specific script.\nEx. so-start filebeat, or so-filebeat-start\n"	
 fi
-
-case $1 in
-   "all") salt-call state.highstate queue=True;;
-   "steno") if docker ps | grep -q so-$1; then printf "\n$1 is already running!\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply pcap queue=True; fi ;; 
-   *) if docker ps | grep -E -q '^so-$1$'; then printf "\n$1 is already running\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply $1 queue=True; fi ;; 
-esac

salt/common/tools/sbin/so-status

@@ -15,7 +15,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 {%- from 'common/maps/so-status.map.jinja' import docker with context %}
-{%- set container_list = docker['containers'] | sort %}
+{%- set container_list = docker['containers'] | sort | unique %}
 
 if ! [ "$(id -u)" = 0 ]; then
    echo "This command must be run as root"
@@ -27,6 +27,7 @@ ERROR_STRING="ERROR"
 SUCCESS_STRING="OK"
 PENDING_STRING="PENDING"
 MISSING_STRING='MISSING'
+CALLER=$(ps -o comm= $PPID)
 declare -a BAD_STATUSES=("removing" "paused" "exited" "dead")
 declare -a PENDING_STATUSES=("paused" "created" "restarting")
 declare -a GOOD_STATUSES=("running")
@@ -71,9 +72,9 @@ compare_lists() {
 # {% endraw %}
 
 create_expected_container_list() {
-    {% for item in container_list%}
+    {% for item in container_list -%}
         expected_container_list+=("{{ item }}")
-    {% endfor %}
+    {% endfor -%}
 }
 
 populate_container_lists() {
@@ -93,7 +94,7 @@ populate_container_lists() {
     for line in "${docker_raw_list[@]}"; do
         container_name="$( echo $line | sed -e 's/Name:\(.*\),State:\(.*\)/\1/' )" # Get value in the first search group (container names)
         container_state="$( echo $line | sed -e 's/Name:\(.*\),State:\(.*\)/\2/' )" # Get value in the second search group (container states)
-        
+
         temp_container_name_list+=( "${container_name}" )
         temp_container_state_list+=( "${container_state}" )
     done
@@ -149,33 +150,78 @@ print_line() {
     printf "%s    \n" " ]"
 }
 
-main() {
-    local focus_color="\e[1;34m"
-    printf "\n"
-    printf "${focus_color}%b\e[0m" "Checking Docker status\n\n"
+non_term_print_line() {
+    local service_name=${1}
+    local service_state="$( parse_status ${2} )"
 
-    systemctl is-active --quiet docker
-    if [[ $? = 0 ]]; then
-        print_line "Docker" "running"
-    else
-        print_line "Docker" "exited"
-    fi
+    local PADDING_CONSTANT=10
 
-    populate_container_lists
-
-    printf "\n"
-    printf "${focus_color}%b\e[0m" "Checking container statuses\n\n"
-
-    local num_containers=${#container_name_list[@]}
-
-    for i in $(seq 0 $(($num_containers - 1 ))); do
-        print_line ${container_name_list[$i]} ${container_state_list[$i]}
+    printf "    $service_name "
+    for i in $(seq 0 $(( 40 - $PADDING_CONSTANT - ${#service_name} - ${#service_state} ))); do
+        printf "-"
     done
+    printf " [ "
+    printf "$service_state"
+    printf "%s    \n" " ]"
+}
 
-    printf "\n"
+main() {
+
+    # if running from salt
+    if [ "$CALLER" == 'salt-call' ] ||  [ "$CALLER" == 'salt-minion' ]; then
+      printf "\n"
+      printf "Checking Docker status\n\n"
+
+      systemctl is-active --quiet docker
+      if [[ $? = 0 ]]; then
+          non_term_print_line "Docker" "running"
+      else
+          non_term_print_line "Docker" "exited"
+      fi
+
+      populate_container_lists
+
+      printf "\n"
+      printf "Checking container statuses\n\n"
+
+      local num_containers=${#container_name_list[@]}
+
+      for i in $(seq 0 $(($num_containers - 1 ))); do
+          non_term_print_line ${container_name_list[$i]} ${container_state_list[$i]}
+      done
+
+      printf "\n"
+    
+    # else if running from a terminal
+    else
+
+      local focus_color="\e[1;34m"
+      printf "\n"
+      printf "${focus_color}%b\e[0m" "Checking Docker status\n\n"
+
+      systemctl is-active --quiet docker
+      if [[ $? = 0 ]]; then
+          print_line "Docker" "running"
+      else
+          print_line "Docker" "exited"
+      fi
+
+      populate_container_lists
+
+      printf "\n"
+      printf "${focus_color}%b\e[0m" "Checking container statuses\n\n"
+
+      local num_containers=${#container_name_list[@]}
+
+      for i in $(seq 0 $(($num_containers - 1 ))); do
+          print_line ${container_name_list[$i]} ${container_state_list[$i]}
+      done
+
+      printf "\n"
+    fi
 }
 
 # {% endraw %}
 
 
-main
+main

salt/common/tools/sbin/so-stop

@@ -19,11 +19,15 @@
 
 . /usr/sbin/so-common
 
-echo $banner
-printf "Stopping $1...\n"
-echo $banner
+if [ $# -ge 1 ]; then
+	echo $banner
+	printf "Stopping $1...\n"
+	echo $banner
 
-case $1 in
-    *) docker stop so-$1 ; docker rm so-$1 ;;
-esac
+	case $1 in
+    		*) docker stop so-$1 ; docker rm so-$1 ;;
+	esac
+else
+	echo -e "\nPlease provide an argument by running like so-stop $component, or by using the component-specific script.\nEx. so-stop filebeat, or so-filebeat-stop\n"	
+fi
 

salt/common/tools/sbin/so-tcpreplay

@@ -15,7 +15,7 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-# Usage: so-tcpreplay "/opt/so/samples/*"
+# Usage: so-tcpreplay "/opt/samples/*"
 
 REPLAY_ENABLED=$(docker images | grep so-tcpreplay)
 REPLAY_RUNNING=$(docker ps | grep so-tcpreplay)

salt/common/tools/sbin/so-test

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Usage: so-test
+
+. /usr/sbin/so-common
+
+REPLAY_ENABLED=$(docker images | grep so-tcpreplay)
+REPLAY_RUNNING=$(docker ps | grep so-tcpreplay)
+
+if  [ "$REPLAY_ENABLED" != "" ] && [ "$REPLAY_RUNNING" != "" ]; then
+   docker cp so-tcpreplay:/opt/samples /opt/samples
+   docker exec -it so-tcpreplay /usr/local/bin/tcpreplay -i bond0 -M10 /opt/samples/*
+   echo
+   echo "PCAP's have been replayed - it is normal to see some warnings."
+   echo
+else
+  echo "Replay functionality not enabled!  Enabling Now...."
+  echo
+  echo "Note that you will need internet access to download the appropriate components"
+  /usr/sbin/so-start tcpreplay
+  echo "Replay functionality enabled.  Replaying PCAPs Now...."
+  docker cp so-tcpreplay:/opt/samples /opt/samples
+  docker exec -it so-tcpreplay /usr/local/bin/tcpreplay -i bond0 -M10 /opt/samples/*
+  echo
+  echo "PCAP's have been replayed - it is normal to see some warnings."
+  echo
+fi
+

salt/common/tools/sbin/so-thehive-user-add

@@ -0,0 +1,52 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <new-user-name>"
+    echo ""
+    echo "Adds a new user to TheHive. The new password will be read from STDIN."
+    exit 1
+}
+
+if [ $# -ne 1 ]; then
+  usage
+fi
+
+USER=$1
+
+THEHIVE_KEY=$(lookup_pillar hivekey)
+THEHIVE_IP=$(lookup_pillar managerip)
+THEHIVE_USER=$USER
+
+# Read password for new user from stdin
+test -t 0
+if [[ $? == 0 ]]; then
+  echo "Enter new password:"
+fi
+read -s THEHIVE_PASS
+
+# Create new user in TheHive
+resp=$(curl -sk -XPOST -H "Authorization: Bearer $THEHIVE_KEY" -H "Content-Type: application/json" "https://$THEHIVE_IP/thehive/api/user" -d "{\"login\" : \"$THEHIVE_USER\",\"name\" : \"$THEHIVE_USER\",\"roles\" : [\"read\",\"alert\",\"write\",\"admin\"],\"preferences\" : \"{}\",\"password\" : \"$THEHIVE_PASS\"}")
+if [[ "$resp" =~ \"status\":\"Ok\" ]]; then
+    echo "Successfully added user to TheHive."
+else
+    echo "Unable to add user to TheHive; user might already exist."
+    echo $resp
+    exit 2
+fi

salt/common/tools/sbin/so-thehive-user-enable

@@ -0,0 +1,57 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+. /usr/sbin/so-common
+
+usage() {
+    echo "Usage: $0 <user-name> <true|false>"
+    echo ""
+    echo "Enables or disables a user in thehive."
+    exit 1
+}
+
+if [ $# -ne 2 ]; then
+  usage
+fi
+
+USER=$1
+
+THEHIVE_KEY=$(lookup_pillar hivekey)
+THEHIVE_IP=$(lookup_pillar managerip)
+THEHIVE_USER=$USER
+
+case "${2^^}" in
+	FALSE | NO | 0)
+		THEHIVE_STATUS=Locked
+		;;
+	TRUE | YES | 1)
+		THEHIVE_STATUS=Ok
+		;;
+	*)
+		usage
+		;;
+esac
+
+resp=$(curl -sk -XPATCH -H "Authorization: Bearer $THEHIVE_KEY" -H "Content-Type: application/json" "https://$THEHIVE_IP/thehive/api/user/${THEHIVE_USER}" -d "{\"status\":\"${THEHIVE_STATUS}\" }")
+if [[ "$resp" =~ \"status\":\"Locked\" || "$resp" =~ \"status\":\"Ok\" ]]; then
+    echo "Successfully updated user in thehive."
+else
+    echo "Failed to update user in thehive."
+    echo "$resp"
+    exit 2
+fi
+    

salt/common/tools/sbin/so-user

@@ -8,26 +8,16 @@
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
 
-got_root() {
-
-  # Make sure you are root
-  if [ "$(id -u)" -ne 0 ]; then
-          echo "This script must be run using sudo!"
-          exit 1
-  fi
-
-}
-
-# Make sure the user is root
-got_root
+. /usr/sbin/so-common
 
 if [[ $# < 1 || $# > 2 ]]; then
-  echo "Usage: $0 <list|add|update|delete|validate|valemail|valpass> [email]"
+  echo "Usage: $0 <list|add|update|enable|disable|validate|valemail|valpass> [email]"
   echo ""
   echo "     list: Lists all user email addresses currently defined in the identity system"
   echo "      add: Adds a new user to the identity system; requires 'email' parameter"
   echo "   update: Updates a user's password; requires 'email' parameter"
-  echo "   delete: Deletes an existing user; requires 'email' parameter"
+  echo "   enable: Enables a user; requires 'email' parameter"
+  echo "  disable: Disables a user; requires 'email' parameter"
   echo " validate: Validates that the given email address and password are acceptable for defining a new user; requires 'email' parameter"
   echo " valemail: Validates that the given email address is acceptable for defining a new user; requires 'email' parameter"
   echo "  valpass: Validates that a password is acceptable for defining a new user"
@@ -74,7 +64,7 @@ function findIdByEmail() {
   email=$1
 
   response=$(curl -Ss ${kratosUrl}/identities)
-  identityId=$(echo "${response}" | jq ".[] | select(.addresses[0].value == \"$email\") | .id")
+  identityId=$(echo "${response}" | jq ".[] | select(.verifiable_addresses[0].value == \"$email\") | .id")
   echo $identityId
 }
 
@@ -124,7 +114,7 @@ function listUsers() {
   response=$(curl -Ss ${kratosUrl}/identities)
   [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
 
-  echo "${response}" | jq -r ".[] | .addresses[0].value" | sort
+  echo "${response}" | jq -r ".[] | .verifiable_addresses[0].value" | sort
 }
 
 function createUser() {
@@ -133,17 +123,8 @@ function createUser() {
   now=$(date -u +%FT%TZ)
   addUserJson=$(cat <<EOF
 {
-  "addresses": [
-    {
-      "expires_at": "2099-01-31T12:00:00Z",
-      "value": "${email}",
-      "verified": true,
-      "verified_at": "${now}",
-      "via": "so-add-user"
-    }
-  ],
   "traits": {"email":"${email}"},
-  "traits_schema_id": "default"
+  "schema_id": "default"
 }
 EOF
   )
@@ -163,6 +144,36 @@ EOF
   updatePassword $identityId
 }
 
+function updateStatus() {
+  email=$1
+  status=$2
+
+  identityId=$(findIdByEmail "$email")
+  [[ ${identityId} == "" ]] && fail "User not found"
+
+  response=$(curl -Ss "${kratosUrl}/identities/$identityId")
+  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
+
+  oldConfig=$(echo "select config from identity_credentials where identity_id=${identityId};" | sqlite3 "$databasePath")
+  if [[ "$status" == "locked" ]]; then
+    config=$(echo $oldConfig | sed -e 's/hashed/locked/')
+    echo "update identity_credentials set config=CAST('${config}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
+    [[ $? != 0 ]] && fail "Unable to lock credential record"
+
+    echo "delete from sessions where identity_id=${identityId};" | sqlite3 "$databasePath"
+    [[ $? != 0 ]] && fail "Unable to invalidate sessions"    
+  else
+    config=$(echo $oldConfig | sed -e 's/locked/hashed/')
+    echo "update identity_credentials set config=CAST('${config}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
+    [[ $? != 0 ]] && fail "Unable to unlock credential record"
+  fi  
+
+  updatedJson=$(echo "$response" | jq ".traits.status = \"$status\" | del(.verifiable_addresses) | del(.id) | del(.schema_url)")
+  response=$(curl -Ss -XPUT ${kratosUrl}/identities/$identityId -d "$updatedJson")
+  [[ $? != 0 ]] && fail "Unable to mark user as locked"
+
+}
+
 function updateUser() {
   email=$1
 
@@ -189,7 +200,9 @@ case "${operation}" in
 
     validateEmail "$email"
     createUser "$email"
-    echo "Successfully added new user"
+    echo "Successfully added new user to SOC"
+    check_container thehive && echo $password | so-thehive-user-add "$email"
+    check_container fleet && echo $password | so-fleet-user-add "$email"
     ;;
 
   "list")
@@ -205,12 +218,34 @@ case "${operation}" in
     echo "Successfully updated user"
     ;;
 
+  "enable")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+
+    updateStatus "$email" 'active'
+    echo "Successfully enabled user"
+    check_container thehive && so-thehive-user-enable "$email" true
+    check_container fleet && so-fleet-user-enable "$email" true   
+    ;;
+
+  "disable")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+
+    updateStatus "$email" 'locked'
+    echo "Successfully disabled user"
+    check_container thehive && so-thehive-user-enable "$email" false
+    check_container fleet && so-fleet-user-enable "$email" false   
+    ;;    
+
   "delete")
     verifyEnvironment
     [[ "$email" == "" ]] && fail "Email address must be provided"
 
     deleteUser "$email"
-    echo "Successfully deleted user"  
+    echo "Successfully deleted user"
+    check_container thehive && so-thehive-user-enable "$email" false
+    check_container fleet && so-fleet-user-enable "$email" false
     ;;
 
   "validate")

salt/common/tools/sbin/so-user-disable

@@ -0,0 +1,2 @@
+#!/bin/bash
+so-user disable $*

salt/common/tools/sbin/so-user-enable

@@ -0,0 +1,2 @@
+#!/bin/bash
+so-user enable $*

salt/common/tools/sbin/so-wazuh-agent-manage

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if docker ps |grep so-wazuh >/dev/null 2>&1; then
+  docker exec -it so-wazuh /var/ossec/bin/manage_agents "$@"
+else
+  echo "Wazuh manager is not running.  Please start it with so-wazuh-start."
+fi

salt/common/tools/sbin/so-wazuh-agent-upgrade

@@ -0,0 +1,22 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+if docker ps |grep so-wazuh >/dev/null 2>&1; then
+  docker exec -it so-wazuh /var/ossec/bin/agent_upgrade "$@"
+else
+  echo "Wazuh manager is not running.  Please start it with so-wazuh-start."
+fi

salt/common/tools/sbin/so-yara-update

@@ -14,10 +14,10 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
+{%- set ISAIRGAP = salt['pillar.get']('global:airgap', 'False') %}
 
-clone_dir="/tmp"
 output_dir="/opt/so/saltstack/default/salt/strelka/rules"
-#mkdir -p $output_dir
+mkdir -p $output_dir
 repos="$output_dir/repos.txt"
 ignorefile="$output_dir/ignore.txt"
 
@@ -25,8 +25,70 @@ deletecounter=0
 newcounter=0
 updatecounter=0
 
-gh_status=$(curl -s -o /dev/null -w "%{http_code}" http://github.com)
+{% if ISAIRGAP is sameas true %}
 
+
+clone_dir="/nsm/repo/rules/strelka"
+repo_name="signature-base"
+mkdir -p /opt/so/saltstack/default/salt/strelka/rules/signature-base
+
+[ -f $clone_dir/LICENSE ] && cp $clone_dir/$repo_name/LICENSE $output_dir/$repo_name
+
+# Copy over rules
+for i in $(find $clone_dir/yara -name "*.yar*"); do
+  rule_name=$(echo $i | awk -F '/' '{print $NF}')
+  repo_sum=$(sha256sum $i | awk '{print $1}')
+
+  # Check rules against those in ignore list -- don't copy if ignored.
+  if ! grep -iq $rule_name $ignorefile; then
+    existing_rules=$(find $output_dir/$repo_name/ -name $rule_name | wc -l)
+
+    # For existing rules, check to see if they need to be updated, by comparing checksums
+    if [ $existing_rules -gt 0 ];then
+      local_sum=$(sha256sum $output_dir/$repo_name/$rule_name | awk '{print $1}')
+      if [ "$repo_sum" != "$local_sum" ]; then
+        echo "Checksums do not match!"
+        echo "Updating $rule_name..."
+        cp $i $output_dir/$repo_name;
+        ((updatecounter++))
+      fi
+    else
+      # If rule doesn't exist already, we'll add it
+      echo "Adding new rule: $rule_name..."
+      cp $i $output_dir/$repo_name
+      ((newcounter++))
+    fi
+  fi;
+done
+
+# Check to see if we have any old rules that need to be removed
+for i in $(find $output_dir/$repo_name -name "*.yar*" | awk -F '/' '{print $NF}'); do
+  is_repo_rule=$(find $clone_dir -name "$i" | wc -l)
+  if [ $is_repo_rule -eq 0 ]; then
+    echo "Could not find $i in source $repo_name repo...removing from $output_dir/$repo_name..."
+    rm $output_dir/$repo_name/$i
+    ((deletecounter++))
+  fi
+done
+
+echo "Done!"
+
+  if [ "$newcounter" -gt 0 ];then
+    echo "$newcounter new rules added."
+  fi
+
+  if [ "$updatecounter" -gt 0 ];then
+    echo "$updatecounter rules updated."
+  fi
+
+  if [ "$deletecounter" -gt 0 ];then
+    echo "$deletecounter rules removed because they were deprecated or don't exist in the source repo."
+  fi
+
+{% else %}
+
+gh_status=$(curl -s -o /dev/null -w "%{http_code}" http://github.com)
+clone_dir="/tmp"
 if [ "$gh_status" == "200" ]  || [ "$gh_status" == "301" ]; then
 
   while IFS= read -r repo; do
@@ -68,7 +130,7 @@ if [ "$gh_status" == "200" ]  || [ "$gh_status" == "301" ]; then
         fi
       fi;
     done
-    
+
     # Check to see if we have any old rules that need to be removed
     for i in $(find $output_dir/$repo_name -name "*.yar*" | awk -F '/' '{print $NF}'); do
       is_repo_rule=$(find $clone_dir/$repo_name -name "$i" | wc -l)
@@ -100,3 +162,4 @@ else
   echo "No connectivity to Github...exiting..."
   exit 1
 fi
+{%- endif -%}

salt/common/tools/sbin/soup

@@ -18,13 +18,18 @@
 . /usr/sbin/so-common
 UPDATE_DIR=/tmp/sogh/securityonion
 INSTALLEDVERSION=$(cat /etc/soversion)
-default_salt_dir=/opt/so/saltstack/default
+INSTALLEDSALTVERSION=$(salt --versions-report | grep Salt: | awk {'print $2'})
+DEFAULT_SALT_DIR=/opt/so/saltstack/default
+BATCHSIZE=5
+SOUP_LOG=/root/soup.log
+exec 3>&1 1>${SOUP_LOG} 2>&1
 
 manager_check() {
   # Check to see if this is a manager
   MANAGERCHECK=$(cat /etc/salt/grains | grep role | awk '{print $2}')
-  if [[ "$MANAGERCHECK" =~ ^('so-eval'|'so-manager'|'so-standalone'|'so-managersearch')$ ]]; then
-    echo "This is a manager. We can proceed"
+  if [[ "$MANAGERCHECK" =~ ^('so-eval'|'so-manager'|'so-standalone'|'so-managersearch'|'so-import')$ ]]; then
+    echo "This is a manager. We can proceed."
+    MINIONID=$(salt-call grains.get id --out=txt|awk -F: {'print $2'}|tr -d ' ')
   else
     echo "Please run soup on the manager. The manager controls all updates."
     exit 0
@@ -33,7 +38,8 @@ manager_check() {
 
 clean_dockers() {
   # Place Holder for cleaning up old docker images
-  echo ""
+  echo "Trying to clean up old dockers."
+  docker system prune -a -f
 }
 
 clone_to_tmp() {
@@ -58,28 +64,157 @@ clone_to_tmp() {
 copy_new_files() {
   # Copy new files over to the salt dir
   cd /tmp/sogh/securityonion
-  rsync -a salt $default_salt_dir/
-  rsync -a pillar $default_salt_dir/
-  chown -R socore:socore $default_salt_dir/
-  chmod 755 $default_salt_dir/pillar/firewall/addfirewall.sh
+  rsync -a salt $DEFAULT_SALT_DIR/
+  rsync -a pillar $DEFAULT_SALT_DIR/
+  chown -R socore:socore $DEFAULT_SALT_DIR/
+  chmod 755 $DEFAULT_SALT_DIR/pillar/firewall/addfirewall.sh
   cd /tmp
 }
 
+detect_os() {
+	# Detect Base OS
+	echo "Determining Base OS." >> "$SOUP_LOG" 2>&1
+	if [ -f /etc/redhat-release ]; then
+		OS="centos"
+	elif [ -f /etc/os-release ]; then
+		OS="ubuntu"
+	fi
+	echo "Found OS: $OS" >> "$SOUP_LOG" 2>&1
+}
+
 highstate() {
   # Run a highstate but first cancel a running one.
   salt-call saltutil.kill_all_jobs
-  salt-call state.highstate
+  salt-call state.highstate -l info
+}
+
+masterlock() {
+  echo "Locking Salt Master"
+  if [[ "$INSTALLEDVERSION" =~ rc.1 ]]; then
+    TOPFILE=/opt/so/saltstack/default/salt/top.sls
+    BACKUPTOPFILE=/opt/so/saltstack/default/salt/top.sls.backup
+    mv -v $TOPFILE $BACKUPTOPFILE
+    echo "base:" > $TOPFILE
+    echo "  $MINIONID:" >> $TOPFILE
+    echo "    - ca" >> $TOPFILE
+    echo "    - ssl" >> $TOPFILE
+    echo "    - elasticsearch" >> $TOPFILE
+  fi
+}
+
+masterunlock() {
+  echo "Unlocking Salt Master"
+  if [[ "$INSTALLEDVERSION" =~ rc.1 ]]; then
+    mv -v $BACKUPTOPFILE $TOPFILE
+  fi
+}
+
+playbook() {
+  echo "Applying playbook settings"
+  if [[ "$INSTALLEDVERSION" =~ rc.1 ]]; then
+    salt-call state.apply playbook.db_init
+    rm -f /opt/so/rules/elastalert/playbook/*.yaml
+    so-playbook-ruleupdate >> /root/soup_playbook_rule_update.log 2>&1 &
+  fi
 }
 
 pillar_changes() {
     # This function is to add any new pillar items if needed.
-    echo "Checking to see if pillar changes are needed"
+    echo "Checking to see if pillar changes are needed."
+    
+    [[ "$INSTALLEDVERSION" =~ rc.1 ]] && rc1_to_rc2
+    [[ "$INSTALLEDVERSION" =~ rc.2 ]] && rc2_to_rc3
 
 }
 
+rc1_to_rc2() {
+
+  # Move the static file to global.sls
+  echo "Migrating static.sls to global.sls"
+  mv -v /opt/so/saltstack/local/pillar/static.sls /opt/so/saltstack/local/pillar/global.sls >> "$SOUP_LOG" 2>&1
+  sed -i '1c\global:' /opt/so/saltstack/local/pillar/global.sls >> "$SOUP_LOG" 2>&1
+
+  # Moving baseurl from minion sls file to inside global.sls
+  local line=$(grep '^  url_base:' /opt/so/saltstack/local/pillar/minions/$MINIONID.sls)
+  sed -i '/^  url_base:/d' /opt/so/saltstack/local/pillar/minions/$MINIONID.sls;
+  sed -i "/^global:/a \\$line" /opt/so/saltstack/local/pillar/global.sls;
+
+  # Adding play values to the global.sls
+  local HIVEPLAYSECRET=$(tr -dc 'a-zA-Z0-9' < /dev/urandom  | fold -w 20 | head -n 1)
+  local CORTEXPLAYSECRET=$(tr -dc 'a-zA-Z0-9' < /dev/urandom  | fold -w 20 | head -n 1)
+  sed -i "/^global:/a \\  hiveplaysecret: $HIVEPLAYSECRET" /opt/so/saltstack/local/pillar/global.sls;
+  sed -i "/^global:/a \\  cortexplaysecret: $CORTEXPLAYSECRET" /opt/so/saltstack/local/pillar/global.sls;
+
+  # Move storage nodes to hostname for SSL
+  # Get a list we can use:
+  grep -A1 searchnode /opt/so/saltstack/local/pillar/data/nodestab.sls | grep -v '\-\-' | sed '$!N;s/\n/ /' | awk '{print $1,$3}' | awk '/_searchnode:/{gsub(/\_searchnode:/, "_searchnode"); print}' >/tmp/nodes.txt
+  # Remove the nodes from cluster settings
+  while read p; do
+  local NAME=$(echo $p | awk '{print $1}')
+  local IP=$(echo $p | awk '{print $2}')
+  echo "Removing the old cross cluster config for $NAME"
+  curl -XPUT -H 'Content-Type: application/json' http://localhost:9200/_cluster/settings -d '{"persistent":{"cluster":{"remote":{"'$NAME'":{"skip_unavailable":null,"seeds":null}}}}}'
+  done </tmp/nodes.txt
+  # Add the nodes back using hostname
+  while read p; do
+      local NAME=$(echo $p | awk '{print $1}')
+      local EHOSTNAME=$(echo $p | awk -F"_" '{print $1}')
+	    local IP=$(echo $p | awk '{print $2}')
+      echo "Adding the new cross cluster config for $NAME"
+      curl -XPUT http://localhost:9200/_cluster/settings -H'Content-Type: application/json' -d '{"persistent": {"search": {"remote": {"'$NAME'": {"skip_unavailable": "true", "seeds": ["'$EHOSTNAME':9300"]}}}}}'
+  done </tmp/nodes.txt
+
+  INSTALLEDVERSION=rc.2
+
+}
+
+rc2_to_rc3() {
+
+  # move location of local.rules
+  cp /opt/so/saltstack/default/salt/idstools/localrules/local.rules /opt/so/saltstack/local/salt/idstools/local.rules
+  
+  if [ -f /opt/so/saltstack/local/salt/idstools/localrules/local.rules ]; then
+    cat /opt/so/saltstack/local/salt/idstools/localrules/local.rules >> /opt/so/saltstack/local/salt/idstools/local.rules
+  fi
+  rm -rf /opt/so/saltstack/local/salt/idstools/localrules
+  rm -rf /opt/so/saltstack/default/salt/idstools/localrules
+
+  # Rename mdengine to MDENGINE
+  sed -i "s/  zeekversion/  mdengine/g" /opt/so/saltstack/local/pillar/global.sls
+  # Enable Strelka Rules
+  sed -i "/  rules:/c\  rules: 1" /opt/so/saltstack/local/pillar/global.sls
+
+}
+
+space_check() {
+  # Check to see if there is enough space
+  CURRENTSPACE=$(df -BG / | grep -v Avail | awk '{print $4}' | sed 's/.$//')
+  if [ "$CURRENTSPACE" -lt "10" ]; then
+      echo "You are low on disk space. Upgrade will try and clean up space.";
+      clean_dockers
+  else
+      echo "Plenty of space for upgrading"
+  fi
+  
+}
+
 update_dockers() {
     # List all the containers
-    if [ $MANAGERCHECK != 'so-helix' ]; then
+    if [ $MANAGERCHECK == 'so-import' ]; then
+      TRUSTED_CONTAINERS=( \
+      "so-idstools" \
+      "so-nginx" \
+      "so-filebeat" \
+      "so-suricata" \
+      "so-soc" \
+      "so-elasticsearch" \
+      "so-kibana" \
+      "so-kratos" \
+      "so-suricata" \
+      "so-registry" \
+      "so-pcaptools" \
+      "so-zeek" )
+    elif [ $MANAGERCHECK != 'so-helix' ]; then
       TRUSTED_CONTAINERS=( \
       "so-acng" \
       "so-thehive-cortex" \
@@ -97,6 +232,7 @@ update_dockers() {
       "so-kibana" \
       "so-kratos" \
       "so-logstash" \
+      "so-minio" \
       "so-mysql" \
       "so-nginx" \
       "so-pcaptools" \
@@ -143,9 +279,9 @@ update_dockers() {
 
 update_version() {
   # Update the version to the latest
-  echo "Updating the version file."
+  echo "Updating the Security Onion version file."
   echo $NEWVERSION > /etc/soversion
-  sed -i "s/$INSTALLEDVERSION/$NEWVERSION/g" /opt/so/saltstack/local/pillar/static.sls
+  sed -i "/  soversion:/c\  soversion: $NEWVERSION" /opt/so/saltstack/local/pillar/global.sls
 }
 
 upgrade_check() {
@@ -154,8 +290,44 @@ upgrade_check() {
     if [ "$INSTALLEDVERSION" == "$NEWVERSION" ]; then
       echo "You are already running the latest version of Security Onion."
       exit 0
+    fi 
+}
+
+upgrade_check_salt() {
+    NEWSALTVERSION=$(grep version: $UPDATE_DIR/salt/salt/master.defaults.yaml | awk {'print $2'})
+    if [ "$INSTALLEDSALTVERSION" == "$NEWSALTVERSION" ]; then
+      echo "You are already running the correct version of Salt for Security Onion."
     else
-      echo "Performing Upgrade from $INSTALLEDVERSION to $NEWVERSION"
+      SALTUPGRADED=True
+      echo "Performing upgrade of Salt from $INSTALLEDSALTVERSION to $NEWSALTVERSION."
+      echo ""
+      # If CentOS
+      if [ "$OS" == "centos" ]; then
+        echo "Removing yum versionlock for Salt."
+        echo ""
+        yum versionlock delete "salt-*"
+        echo "Updating Salt packages and restarting services."
+        echo ""
+        sh $UPDATE_DIR/salt/salt/scripts/bootstrap-salt.sh -F -M -x python3 stable "$NEWSALTVERSION"
+        echo "Applying yum versionlock for Salt."
+        echo ""
+        yum versionlock add "salt-*"
+      # Else do Ubuntu things
+      elif [ "$OS" == "ubuntu" ]; then
+        echo "Removing apt hold for Salt."
+        echo ""
+        apt-mark unhold "salt-common"
+        apt-mark unhold "salt-master"
+        apt-mark unhold "salt-minion"
+        echo "Updating Salt packages and restarting services."
+        echo ""
+        sh $UPDATE_DIR/salt/salt/scripts/bootstrap-salt.sh -F -M -x python3 stable "$NEWSALTVERSION"
+        echo "Applying apt hold for Salt."
+        echo ""
+        apt-mark hold "salt-common"
+        apt-mark hold "salt-master"
+        apt-mark hold "salt-minion"
+      fi
     fi 
 }
 
@@ -167,41 +339,109 @@ verify_latest_update_script() {
       echo "This version of the soup script is up to date. Proceeding."
     else
       echo "You are not running the latest soup version. Updating soup."
-      cp $UPDATE_DIR/salt/common/tools/sbin/soup $default_salt_dir/salt/common/tools/sbin/
+      cp $UPDATE_DIR/salt/common/tools/sbin/soup $DEFAULT_SALT_DIR/salt/common/tools/sbin/
       salt-call state.apply common queue=True
       echo ""
-      echo "soup has been updated. Please run soup again"
+      echo "soup has been updated. Please run soup again."
       exit 0
     fi
 }
 
-echo "Checking to see if this is a manager"
+main () {
+while getopts ":b" opt; do
+  case "$opt" in
+    b ) # process option b
+       shift
+       BATCHSIZE=$1
+       if ! [[ "$BATCHSIZE" =~ ^[0-9]+$ ]]; then
+         echo "Batch size must be a number greater than 0."
+         exit 1
+       fi
+      ;;
+    \? ) echo "Usage: cmd [-b]"
+      ;;
+  esac
+done
+
+echo "Checking to see if this is a manager."
+echo ""
 manager_check
-echo "Cloning latest code to a temporary location"
+echo "Found that Security Onion $INSTALLEDVERSION is currently installed."
+echo ""
+detect_os
+echo ""
+echo "Cloning Security Onion github repo into $UPDATE_DIR."
 clone_to_tmp
 echo ""
-echo "Verifying we have the latest script"
+echo "Verifying we have the latest soup script."
 verify_latest_update_script
 echo ""
-echo "Let's see if we need to update"
+
+echo "Let's see if we need to update Security Onion."
 upgrade_check
+space_check
+
 echo ""
-echo "Making pillar changes"
+echo "Performing upgrade from Security Onion $INSTALLEDVERSION to Security Onion $NEWVERSION."
+echo ""
+echo "Stopping Salt Minion service."
+systemctl stop salt-minion
+echo ""
+echo "Stopping Salt Master service."
+systemctl stop salt-master
+echo ""
+echo "Checking for Salt Master and Minion updates."
+upgrade_check_salt
+
+
+echo "Making pillar changes."
 pillar_changes
 echo ""
-echo "Cleaning up old dockers"
-clean_dockers
+
 echo ""
-echo "Updating docker to $NEWVERSION"
+echo "Updating dockers to $NEWVERSION."
 update_dockers
+
 echo ""
-echo "Copying new code"
+echo "Copying new Security Onion code from $UPDATE_DIR to $DEFAULT_SALT_DIR."
 copy_new_files
 echo ""
-echo "Updating version"
 update_version
+
 echo ""
-echo "Running a highstate to complete upgrade"
+echo "Locking down Salt Master for upgrade"
+masterlock
+
+echo ""
+echo "Starting Salt Master service."
+systemctl start salt-master
+
+echo ""
+echo "Running a highstate to complete the Security Onion upgrade on this manager. This could take several minutes."
 highstate
 echo ""
 echo "Upgrade from $INSTALLEDVERSION to $NEWVERSION complete."
+
+echo ""
+echo "Stopping Salt Master to remove ACL"
+systemctl stop salt-master
+
+masterunlock
+
+echo ""
+echo "Starting Salt Master service."
+systemctl start salt-master
+highstate
+playbook
+
+SALTUPGRADED="True"
+if [[ "$SALTUPGRADED" == "True" ]]; then
+  echo ""
+  echo "Upgrading Salt on the remaining Security Onion nodes from $INSTALLEDSALTVERSION to $NEWSALTVERSION."
+  salt -C 'not *_eval and not *_helix and not *_manager and not *_managersearch and not *_standalone' -b $BATCHSIZE state.apply salt.minion 
+  echo ""
+fi
+
+}
+
+main "$@" | tee /dev/fd/3

salt/curator/init.sls

@@ -1,5 +1,10 @@
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'curator' in top_states %}
+
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
 {% if grains['role'] in ['so-eval', 'so-node', 'so-managersearch', 'so-heavynode', 'so-standalone'] %}
 # Curator
@@ -131,3 +136,11 @@ so-curator:
 
 # End Curator Cron Jobs
 {% endif %}
+
+{% else %}
+
+curator_state_not_allowed:
+  test.fail_without_changes:
+    - name: curator_state_not_allowed
+
+{% endif %}

salt/deprecated-launcher/init.sls

@@ -1,4 +1,4 @@
-{%- set FLEETSETUP = salt['pillar.get']('static:fleetsetup', '0') -%}
+{%- set FLEETSETUP = salt['pillar.get']('global:fleetsetup', '0') -%}
 
 {%- if FLEETSETUP != 0 %}
 launcherpkg:

salt/docker/init.sls

@@ -1,3 +1,8 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'docker' in top_states %}
+
 installdocker:
   pkg.installed:
     - name: docker-ce
@@ -5,4 +10,12 @@ installdocker:
 # Make sure Docker is running!
 docker:
   service.running:
-    - enable: True
+    - enable: True
+
+{% else %}
+
+docker_state_not_allowed:
+  test.fail_without_changes:
+    - name: docker_state_not_allowed
+
+{% endif %}

salt/domainstats/init.sls

@@ -12,8 +12,12 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
 
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% if 'domainstats' in top_states %}
+
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 
 # Create the group
 dstatsgroup:
@@ -51,3 +55,11 @@ so-domainstats:
     - user: domainstats
     - binds:
       - /opt/so/log/domainstats:/var/log/domain_stats
+
+{% else %}
+
+domainstats_state_not_allowed:
+  test.fail_without_changes:
+    - name: domainstats_state_not_allowed
+
+{% endif %}

salt/elastalert/files/elastalert_config.yaml

@@ -16,12 +16,12 @@ disable_rules_on_error: false
 # How often ElastAlert will query Elasticsearch
 # The unit can be anything from weeks to seconds
 run_every:
-  minutes: 1
+  minutes: 3
 
 # ElastAlert will buffer results from the most recent
 # period of time, in case some log sources are not in real time
 buffer_time:
-  minutes: 1
+  minutes: 10
 
 # The maximum time between queries for ElastAlert to start at the most recently
 # run query. When ElastAlert starts, for each rule, it will search elastalert_metadata
@@ -38,7 +38,7 @@ es_host: {{ esip }}
 es_port: {{ esport }}
 
 # Sets timeout for connecting to and reading from es_host
-es_conn_timeout: 60
+es_conn_timeout: 55
 
 # The maximum number of documents that will be downloaded from Elasticsearch in
 # a single query. The default is 10,000, and if you expect to get near this number,

salt/elastalert/files/modules/so/playbook-es.py

@@ -16,7 +16,7 @@ class PlaybookESAlerter(Alerter):
             today = strftime("%Y.%m.%d", gmtime())
             timestamp = strftime("%Y-%m-%d"'T'"%H:%M:%S", gmtime())
             headers = {"Content-Type": "application/json"}
-            payload = {"rule.name": self.rule['play_title'],"event.severity": self.rule['event.severity'],"kibana_pivot": self.rule['kibana_pivot'],"soc_pivot": self.rule['soc_pivot'],"event.module": self.rule['event.module'],"event.dataset": self.rule['event.dataset'],"play_url": self.rule['play_url'],"sigma_level": self.rule['sigma_level'],"rule.category": self.rule['rule.category'],"data": match, "@timestamp": timestamp}
+            payload = {"rule.name": self.rule['play_title'],"event.severity": self.rule['event.severity'],"kibana_pivot": self.rule['kibana_pivot'],"soc_pivot": self.rule['soc_pivot'],"event.module": self.rule['event.module'],"event.dataset": self.rule['event.dataset'],"play_url": self.rule['play_url'],"sigma_level": self.rule['sigma_level'],"rule.category": self.rule['rule.category'],"event_data": match, "@timestamp": timestamp}
             url = f"http://{self.rule['elasticsearch_host']}/so-playbook-alerts-{today}/_doc/"
             requests.post(url, data=json.dumps(payload), headers=headers, verify=False)
                             

salt/elastalert/files/rules/so/suricata_thehive.yaml

@@ -1,21 +1,17 @@
-{% set es = salt['pillar.get']('static:managerip', '') %}
-{% set hivehost = salt['pillar.get']('static:managerip', '') %}
-{% set hivekey = salt['pillar.get']('static:hivekey', '') %}
-{% set MANAGER = salt['pillar.get']('manager:url_base', '') %}
+{% set es = salt['pillar.get']('global:managerip', '') %}
+{% set hivehost = salt['pillar.get']('global:managerip', '') %}
+{% set hivekey = salt['pillar.get']('global:hivekey', '') %}
+{% set MANAGER = salt['pillar.get']('global:url_base', '') %}
 
 # Elastalert rule to forward Suricata alerts from Security Onion to a specified TheHive instance.
 #
 es_host: {{es}}
 es_port: 9200
 name: Suricata-Alert
-type: frequency
-index: "so-ids-*"
-num_events: 1
-timeframe:
-    minutes: 10
+type: any
+index: "*:so-ids-*"
 buffer_time:
-    minutes: 10
-allow_buffer_time_overlap: true
+    minutes: 5
 query_key: ["rule.uuid","source.ip","destination.ip"]
 realert:
     days: 1
@@ -39,7 +35,7 @@ hive_alert_config:
   title: '{match[rule][name]}'
   type: 'NIDS'
   source: 'SecurityOnion'
-  description: "`SOC Hunt Pivot:` \n\n <https://{{MANAGER}}/#/hunt?q=network.community_id%3A%20%20%22{match[network][community_id]}%22%20%7C%20groupby%20source.ip%20destination.ip,event.module,%20event.dataset>  \n\n `Kibana Dashboard Pivot:` \n\n <https://{{MANAGER}}/kibana/app/kibana#/dashboard/30d0ac90-729f-11ea-8dd2-9d8795a1200b?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'*:so-*',key:network.community_id,negate:!f,params:(query:'{match[network][community_id]}'),type:phrase),query:(match_phrase:(network.community_id:'{match[network][community_id]}')))),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))>  \n\n `IPs: `{match[source][ip]}:{match[source][port]} --> {match[destination][ip]}:{match[destination][port]} \n\n `Signature:`{match[rule][rule]}"
+  description: "`SOC Hunt Pivot:` \n\n <https://{{MANAGER}}/#/hunt?q=network.community_id%3A%20%20%22{match[network][community_id]}%22%20%7C%20groupby%20source.ip%20destination.ip,event.module,%20event.dataset>  \n\n `Kibana Dashboard Pivot:` \n\n <https://{{MANAGER}}/kibana/app/kibana#/dashboard/30d0ac90-729f-11ea-8dd2-9d8795a1200b?_g=(filters:!(('$state':(store:globalState),meta:(alias:!n,disabled:!f,index:'2289a0c0-6970-11ea-a0cd-ffa0f6a1bc29',key:network.community_id,negate:!f,params:(query:'{match[network][community_id]}'),type:phrase),query:(match_phrase:(network.community_id:'{match[network][community_id]}')))),refreshInterval:(pause:!t,value:0),time:(from:now-7d,to:now))>  \n\n `IPs: `{match[source][ip]}:{match[source][port]} --> {match[destination][ip]}:{match[destination][port]} \n\n `Signature:`{match[rule][rule]}"
   severity: 2
   tags: ['{match[rule][uuid]}','{match[source][ip]}','{match[destination][ip]}']
   tlp: 3

salt/elastalert/files/rules/so/wazuh_thehive.yaml

@@ -1,21 +1,17 @@
-{% set es = salt['pillar.get']('static:managerip', '') %}
-{% set hivehost = salt['pillar.get']('static:managerip', '') %}
-{% set hivekey = salt['pillar.get']('static:hivekey', '') %}
-{% set MANAGER = salt['pillar.get']('manager:url_base', '') %}
+{% set es = salt['pillar.get']('global:managerip', '') %}
+{% set hivehost = salt['pillar.get']('global:managerip', '') %}
+{% set hivekey = salt['pillar.get']('global:hivekey', '') %}
+{% set MANAGER = salt['pillar.get']('global:url_base', '') %}
 
 # Elastalert rule to forward high level Wazuh alerts from Security Onion to a specified TheHive instance.
 #
 es_host: {{es}}
 es_port: 9200
 name: Wazuh-Alert
-type: frequency
-index: "so-ossec-*"
-num_events: 1
-timeframe:
-    minutes: 10
+type: any
+index: "*:so-ossec-*"
 buffer_time:
-    minutes: 10
-allow_buffer_time_overlap: true
+    minutes: 5
 realert:
     days: 1
 filter:

salt/elastalert/init.sls

@@ -12,9 +12,16 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'elastalert' in top_states %}
+
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
+{%- set MANAGER_URL = salt['pillar.get']('global:url_base', '') %}
+{%- set MANAGER_IP = salt['pillar.get']('global:managerip', '') %}
 
 {% if grains['role'] in ['so-eval','so-managersearch', 'so-manager', 'so-standalone'] %}
   {% set esalert = salt['pillar.get']('manager:elastalert', '1') %}
@@ -100,6 +107,12 @@ elastaconf:
     - group: 933
     - template: jinja
 
+wait_for_elasticsearch:
+  module.run:
+    - http.wait_for_successful_query:
+      - url: 'http://{{MANAGER}}:9200/_cat/indices/.kibana*'
+      - wait_for: 180
+
 so-elastalert:
   docker_container.running:
     - image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elastalert:{{ VERSION }}
@@ -112,5 +125,16 @@ so-elastalert:
       - /opt/so/log/elastalert:/var/log/elastalert:rw
       - /opt/so/conf/elastalert/modules/:/opt/elastalert/modules/:ro
       - /opt/so/conf/elastalert/elastalert_config.yaml:/opt/config/elastalert_config.yaml:ro
-
+    - extra_hosts:
+      - {{MANAGER_URL}}:{{MANAGER_IP}}
+    - require:
+      - module: wait_for_elasticsearch
 {% endif %}
+
+{% else %}
+
+elastalert_state_not_allowed:
+  test.fail_without_changes:
+    - name: elastalert_state_not_allowed
+
+{% endif %}

salt/elasticsearch/files/elasticsearch.yml

@@ -5,6 +5,7 @@
 {%- set ESCLUSTERNAME = salt['pillar.get']('elasticsearch:esclustername', '') %}
 {%- endif %}
 {%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
+{% set FEATURES = salt['pillar.get']('elastic:features', False) %}
 cluster.name: "{{ ESCLUSTERNAME }}"
 network.host: 0.0.0.0
 
@@ -16,12 +17,30 @@ discovery.zen.minimum_master_nodes: 1
 path.logs: /var/log/elasticsearch
 action.destructive_requires_name: true
 transport.bind_host: 0.0.0.0
-transport.publish_host: {{ NODEIP }}
+transport.publish_host: {{ grains.host }}
 transport.publish_port: 9300
 cluster.routing.allocation.disk.threshold_enabled: true
 cluster.routing.allocation.disk.watermark.low: 95%
 cluster.routing.allocation.disk.watermark.high: 98%
 cluster.routing.allocation.disk.watermark.flood_stage: 98%
+{%- if FEATURES is sameas true %}
+#xpack.security.enabled: false
+#xpack.security.http.ssl.enabled: false
+#xpack.security.transport.ssl.enabled: false
+#xpack.security.http.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
+#xpack.security.http.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
+#xpack.security.http.ssl.certificate_authorities: /usr/share/elasticsearch/config/ca.crt
+#xpack.security.transport.ssl.key: /usr/share/elasticsearch/config/elasticsearch.key
+#xpack.security.transport.ssl.certificate: /usr/share/elasticsearch/config/elasticsearch.crt
+#xpack.security.transport.ssl.certificate_authorities: /usr/share/elasticsearch/config/ca.crt
+#xpack.security.transport.ssl.verification_mode: none
+#xpack.security.http.ssl.client_authentication: none
+#xpack.security.authc:
+#  anonymous:
+#    username: anonymous_user 
+#    roles: superuser 
+#    authz_exception: true 
+{%- endif %}
 node.attr.box_type: {{ NODE_ROUTE_TYPE }}
 node.name: {{ ESCLUSTERNAME }}
 script.max_compilations_rate: 1000/1m

salt/elasticsearch/files/ingest/beats.common

@@ -1,53 +1,8 @@
 {
   "description" : "beats.common",
   "processors" : [
-    {"community_id":   {"if": "ctx.winlog.event_data?.Protocol != null", "field":["winlog.event_data.SourceIp","winlog.event_data.SourcePort","winlog.event_data.DestinationIp","winlog.event_data.DestinationPort","winlog.event_data.Protocol"],"target_field":"network.community_id"}},    
-    { "set":           { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational'",   "field": "event.module", "value": "sysmon", "override": true }  },
-    { "set":           { "if": "ctx.winlog?.channel != null",   "field": "event.module", "value": "windows_eventlog", "override": false, "ignore_failure": true }  },
-    { "set":           { "if": "ctx.agent?.type != null",   "field": "module", "value": "{{agent.type}}", "override": true }  }, 
-    { "set":           { "if": "ctx.winlog?.channel != 'Microsoft-Windows-Sysmon/Operational'",   "field": "event.dataset", "value": "{{winlog.channel}}", "override": true }  },
-    { "set":           { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 3",   "field": "event.category", "value": "host,process,network", "override": true }  },
-    { "set":           { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 1",   "field": "event.category", "value": "host,process", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 1",   "field": "event.dataset", "value": "process_creation", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 2",   "field": "event.dataset", "value": "process_changed_file", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 3",   "field": "event.dataset", "value": "network_connection", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 5",   "field": "event.dataset", "value": "process_terminated", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 6",   "field": "event.dataset", "value": "driver_loaded", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 7",   "field": "event.dataset", "value": "image_loaded", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 8",   "field": "event.dataset", "value": "create_remote_thread", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 9",   "field": "event.dataset", "value": "raw_file_access_read", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 10",   "field": "event.dataset", "value": "process_access", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 11",   "field": "event.dataset", "value": "file_create", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 12",   "field": "event.dataset", "value": "registry_create_delete", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 13",   "field": "event.dataset", "value": "registry_value_set", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 14",   "field": "event.dataset", "value": "registry_key_value_rename", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 15",   "field": "event.dataset", "value": "file_create_stream_hash", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 16",   "field": "event.dataset", "value": "config_change", "override": true }  },
-    { "set":          { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 22",   "field": "event.dataset", "value": "dns_query", "override": true }  },
-    { "rename": 	   { "field": "agent.hostname", 			            "target_field": "agent.name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.SubjectUserName", 			  "target_field": "user.name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.DestinationHostname", 	  "target_field": "destination.hostname",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.DestinationIp", 		      "target_field": "destination.ip",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.DestinationPort", 	      "target_field": "destination.port",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.Image", 			            "target_field": "process.executable",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.ProcessID", 			        "target_field": "process.pid",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.ProcessGuid", 			      "target_field": "process.entity_id",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.CommandLine", 			      "target_field": "process.command_line",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.CurrentDirectory", 			"target_field": "process.working_directory",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.Description", 			      "target_field": "process.pe.description",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.Product", 			          "target_field": "process.pe.product",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.OriginalFileName", 			"target_field": "process.pe.original_file_name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.FileVersion", 			      "target_field": "process.pe.file_version",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.ParentCommandLine", 			"target_field": "process.parent.command_line",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.ParentImage", 			      "target_field": "process.parent.executable",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.ParentProcessGuid", 			"target_field": "process.parent.entity_id",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.ParentProcessId", 			  "target_field": "process.ppid",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.Protocol", 	              "target_field": "network.transport",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.User", 			            "target_field": "user.name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.SourceHostname", 	      "target_field": "source.hostname",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.SourceIp", 		          "target_field": "source.ip",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "winlog.event_data.SourcePort", 		        "target_field": "source.port",		"ignore_missing": true 	} },   
-    { "rename": 	   { "field": "winlog.event_data.targetFilename",		      "target_field": "file.target",	"ignore_missing": true 	} },
+    { "pipeline":      { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational'",   "name": "sysmon"  }  },
+    { "pipeline":      { "if": "ctx.winlog?.channel != 'Microsoft-Windows-Sysmon/Operational'",  "name":"win.eventlogs" }  },
     { "pipeline":    { "name": "common" } }
   ]
 }

salt/elasticsearch/files/ingest/common

@@ -42,13 +42,14 @@
     { "set":         { "if": "ctx.event?.severity == 3",   "field": "event.severity_label", "value": "high", "override": true }  },
     { "set":         { "if": "ctx.event?.severity == 4",   "field": "event.severity_label", "value": "critical", "override": true }  },
     { "rename":         { "field": "module",              "target_field": "event.module",                  "ignore_failure": true, "ignore_missing": true  } },
-    { "rename":         { "field": "dataset",              "target_field": "event.dataset",                  "ignore_failure": true, "ignore_missing": true  } },
-    { "rename":         { "field": "category",              "target_field": "event.category",                  "ignore_missing": true  } },
+    { "rename":         { "field": "dataset",             "target_field": "event.dataset",                "ignore_failure": true, "ignore_missing": true  } },
+    { "rename":         { "field": "category",            "target_field": "event.category",              "ignore_failure": true, "ignore_missing": true  } },
     { "rename":         { "field": "message2.community_id", "target_field": "network.community_id",  "ignore_failure": true,  "ignore_missing": true  } },
     { "lowercase":         { "field": "event.dataset", "ignore_failure": true,  "ignore_missing": true  } },    
     { "convert":         { "field": "destination.port", "type": "integer",  "ignore_failure": true,  "ignore_missing": true  } },
     { "convert":         { "field": "source.port", "type": "integer",  "ignore_failure": true,  "ignore_missing": true  } },
     { "convert":         { "field": "log.id.uid", "type": "string",  "ignore_failure": true,  "ignore_missing": true  } },
+    { "convert":         { "field": "agent.id", "type": "string",  "ignore_failure": true,  "ignore_missing": true  } },
     { "convert":         { "field": "event.severity", "type": "integer",  "ignore_failure": true,  "ignore_missing": true  } },
     { 
         "remove": {

salt/elasticsearch/files/ingest/import.wel

@@ -0,0 +1,9 @@
+{
+  "description" : "import.wel",
+  "processors" : [
+    { "remove":         { "field": ["event.created","timestamp", "winlog.event_data.UtcTime", "event_record_id"],     "ignore_failure": true                                               } },
+    { "pipeline":      { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational'",   "name": "sysmon"  }  },
+    { "pipeline":      { "if": "ctx.winlog?.channel != 'Microsoft-Windows-Sysmon/Operational'",  "name":"win.eventlogs" }  },
+    { "pipeline":    { "name": "common" } }
+  ]
+}

salt/elasticsearch/files/ingest/osquery.query_result

@@ -2,78 +2,24 @@
   "description" : "osquery",
   "processors" : [
     { "json":        { "field": "message",                                            "target_field": "message2",             "ignore_failure": true  } },
+
     { "gsub":        { "field": "message2.columns.data",                              "pattern": "\\\\xC2\\\\xAE",             "replacement": "", "ignore_missing": true  } },
-    { "json":        { "field": "message2.columns.data",                              "target_field": "message2.columns.winlog",             "ignore_failure": true  } },
+    { "rename":      { "if": "ctx.message2.columns?.eventid != null", "field": "message2.columns",  "target_field": "winlog",       "ignore_missing": true  } },
+    { "json":        { "field": "winlog.data",                              "target_field": "temp",             "ignore_failure": true  } },
+    { "rename":      { "field": "temp.Data",               "target_field": "winlog.event_data",       "ignore_missing": true  } },   
+    { "rename":      { "field": "winlog.source",               "target_field": "winlog.channel",       "ignore_missing": true  } },    
+    { "rename":      { "field": "winlog.eventid",               "target_field": "winlog.event_id",       "ignore_missing": true  } },   
+    { "pipeline":      { "if": "ctx.winlog?.channel == 'Microsoft-Windows-Sysmon/Operational'",   "name": "sysmon"  }  },
+    { "pipeline":      { "if": "ctx.winlog?.channel != 'Microsoft-Windows-Sysmon/Operational'",  "name":"win.eventlogs" }  },
+
     {
       "script": {
         "lang": "painless",
         "source": "def dict = ['result': new HashMap()]; for (entry in ctx['message2'].entrySet()) { dict['result'][entry.getKey()] = entry.getValue(); } ctx['osquery'] = dict; "
       }
     },
-    { "rename":      { "field": "osquery.result.hostIdentifier",                            "target_field": "osquery.result.host_identifier",              "ignore_missing": true  } },
-    { "rename":      { "field": "osquery.result.calendarTime",                              "target_field": "osquery.result.calendar_time",              "ignore_missing": true  } },
-    { "rename":      { "field": "osquery.result.unixTime",                                  "target_field": "osquery.result.unix_time",                "ignore_missing": true  } },    
-    { "json":        { "field": "message",                                                  "target_field": "message3",             "ignore_failure": true  } },
-    { "gsub":        { "field": "message3.columns.data",                                    "pattern": "\\\\xC2\\\\xAE",             "replacement": "", "ignore_missing": true  } },
-    { "json":        { "field": "message3.columns.data",                                    "target_field": "message3.columns.winlog",             "ignore_failure": true  } },
-    { "rename":      { "field": "message3.columns.username",                                "target_field": "user.name",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.uid",                                     "target_field": "user.uid",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.gid",                                     "target_field": "user.gid",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.shell",                                   "target_field": "user.shell",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.cmdline",                                 "target_field": "process.command_line",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.pid",                                     "target_field": "process.pid",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.parent",                                  "target_field": "process.ppid",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.cwd",                                     "target_field": "process.working_directory",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.community_id",                            "target_field": "network.community_id",   "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.local_address",                           "target_field": "local.ip",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.local_port",                              "target_field": "local.port",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.remote_address",                          "target_field": "remote.ip",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.remote_port",                             "target_field": "remote.port",              "ignore_missing": true  } },
-    { "rename":      { "field": "message3.columns.process_name",                            "target_field": "process.name",              "ignore_missing": true  } },
-    { "rename": 	   { "field": "message3.columns.eventid", 			                          "target_field": "event.code",		"ignore_missing": true 	} },
-    { "set":           { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational'",   "field": "event.module", "value": "sysmon", "override": true }  },
-    { "set":           { "if": "ctx.message3.columns?.source != null",   "field": "event.module", "value": "windows_eventlog", "override": false, "ignore_failure": true }  },
-    { "set":           { "if": "ctx.message3.columns?.source != 'Microsoft-Windows-Sysmon/Operational'",   "field": "event.dataset", "value": "{{message3.columns.source}}", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 1",   "field": "event.dataset", "value": "process_creation", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 2",   "field": "event.dataset", "value": "process_changed_file", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 3",   "field": "event.dataset", "value": "network_connection", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 5",   "field": "event.dataset", "value": "process_terminated", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 6",   "field": "event.dataset", "value": "driver_loaded", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 7",   "field": "event.dataset", "value": "image_loaded", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 8",   "field": "event.dataset", "value": "create_remote_thread", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 9",   "field": "event.dataset", "value": "raw_file_access_read", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 10",   "field": "event.dataset", "value": "process_access", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 11",   "field": "event.dataset", "value": "file_create", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 12",   "field": "event.dataset", "value": "registry_create_delete", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 13",   "field": "event.dataset", "value": "registry_value_set", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 14",   "field": "event.dataset", "value": "registry_key_value_rename", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 15",   "field": "event.dataset", "value": "file_create_stream_hash", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 16",   "field": "event.dataset", "value": "config_change", "override": true }  },
-    { "set":          { "if": "ctx.message3.columns?.source == 'Microsoft-Windows-Sysmon/Operational' && ctx.event?.code == 22",   "field": "event.dataset", "value": "dns_query", "override": true }  },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.SubjectUserName", 			  "target_field": "user.name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.destinationHostname", 	  "target_field": "destination.hostname",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.destinationIp", 		      "target_field": "destination.ip",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.destinationPort", 	      "target_field": "destination.port",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.Image", 			            "target_field": "process.executable",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.ProcessID", 			        "target_field": "process.pid",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.ProcessGuid", 			      "target_field": "process.entity_id",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.CommandLine", 			      "target_field": "process.command_line",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.CurrentDirectory", 			"target_field": "process.working_directory",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.Description", 			      "target_field": "process.pe.description",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.Product", 			          "target_field": "process.pe.product",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.OriginalFileName", 			"target_field": "process.pe.original_file_name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.FileVersion", 			      "target_field": "process.pe.file_version",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentCommandLine", 			"target_field": "process.parent.command_line",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentImage", 			      "target_field": "process.parent.executable",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentProcessGuid", 			"target_field": "process.parent.entity_id",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentProcessId", 			  "target_field": "process.ppid",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.User", 			            "target_field": "user.name",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.parentImage", 		        "target_field": "parent_image_path",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.sourceHostname", 	      "target_field": "source.hostname",	"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.sourceIp", 		          "target_field": "source_ip",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.sourcePort", 		        "target_field": "source.port",		"ignore_missing": true 	} },
-    { "rename": 	   { "field": "message3.columns.winlog.EventData.targetFilename",		      "target_field": "file.target",	"ignore_missing": true 	} },
-    { "remove":      { "field": [ "message3"],                                              "ignore_failure": false } },
+    { "set":           { "field": "event.module", "value": "osquery", "override": false }  },
+    { "set":           { "field": "event.dataset", "value": "{{osquery.result.name}}", "override": false}  },
     { "pipeline":    { "name": "common" } }
   ]
 }

salt/elasticsearch/files/ingest/sysmon

@@ -0,0 +1,54 @@
+{
+  "description" : "sysmon",
+  "processors" : [
+    {"community_id":   {"if": "ctx.winlog.event_data?.Protocol != null", "field":["winlog.event_data.SourceIp","winlog.event_data.SourcePort","winlog.event_data.DestinationIp","winlog.event_data.DestinationPort","winlog.event_data.Protocol"],"target_field":"network.community_id"}},    
+    { "set":           { "field": "event.code", "value": "{{winlog.event_id}}", "override": true }  },     
+    { "set":           { "field": "event.module", "value": "sysmon", "override": true }  },   
+    { "set":           { "if": "ctx.winlog?.computer_name != null",   "field": "observer.name", "value": "{{winlog.computer_name}}", "override": true }  },
+    { "set":           { "if": "ctx.event?.code == '3'",   "field": "event.category", "value": "host,process,network", "override": true }  },
+    { "set":           { "if": "ctx.event?.code == '1'",   "field": "event.category", "value": "host,process", "override": true }  },
+    { "set":           { "if": "ctx.event?.code == '5'",   "field": "event.category", "value": "host,process", "override": true }  },
+    { "set":           { "if": "ctx.event?.code == '6'",   "field": "event.category", "value": "host,driver", "override": true }  },
+    { "set":           { "if": "ctx.event?.code == '22'",   "field": "event.category", "value": "network", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '1'",   "field": "event.dataset", "value": "process_creation", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '2'",   "field": "event.dataset", "value": "process_changed_file", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '3'",   "field": "event.dataset", "value": "network_connection", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '5'",   "field": "event.dataset", "value": "process_terminated", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '6'",   "field": "event.dataset", "value": "driver_loaded", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '7'",   "field": "event.dataset", "value": "image_loaded", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '8'",   "field": "event.dataset", "value": "create_remote_thread", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '9'",   "field": "event.dataset", "value": "raw_file_access_read", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '10'",   "field": "event.dataset", "value": "process_access", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '11'",   "field": "event.dataset", "value": "file_create", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '12'",   "field": "event.dataset", "value": "registry_create_delete", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '13'",   "field": "event.dataset", "value": "registry_value_set", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '14'",   "field": "event.dataset", "value": "registry_key_value_rename", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '15'",   "field": "event.dataset", "value": "file_create_stream_hash", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '16'",   "field": "event.dataset", "value": "config_change", "override": true }  },
+    { "set":          { "if": "ctx.event?.code == '22'",   "field": "event.dataset", "value": "dns_query", "override": true }  },
+    { "rename": 	   { "field": "winlog.event_data.SubjectUserName", 			  "target_field": "user.name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.DestinationHostname", 	  "target_field": "destination.hostname",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.DestinationIp", 		      "target_field": "destination.ip",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.DestinationPort", 	      "target_field": "destination.port",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.Image", 			            "target_field": "process.executable",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.ProcessID", 			        "target_field": "process.pid",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.ProcessGuid", 			      "target_field": "process.entity_id",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.CommandLine", 			      "target_field": "process.command_line",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.CurrentDirectory", 			"target_field": "process.working_directory",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.Description", 			      "target_field": "process.pe.description",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.Product", 			          "target_field": "process.pe.product",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.Company", 			          "target_field": "process.pe.company",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.OriginalFileName", 			"target_field": "process.pe.original_file_name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.FileVersion", 			      "target_field": "process.pe.file_version",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.ParentCommandLine", 			"target_field": "process.parent.command_line",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.ParentImage", 			      "target_field": "process.parent.executable",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.ParentProcessGuid", 			"target_field": "process.parent.entity_id",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.ParentProcessId", 			  "target_field": "process.ppid",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.Protocol", 	              "target_field": "network.transport",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.User", 			            "target_field": "user.name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.SourceHostname", 	      "target_field": "source.hostname",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.SourceIp", 		          "target_field": "source.ip",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.SourcePort", 		        "target_field": "source.port",		"ignore_missing": true 	} },   
+    { "rename": 	   { "field": "winlog.event_data.targetFilename",		      "target_field": "file.target",	"ignore_missing": true 	} }
+  ]
+}

salt/elasticsearch/files/ingest/win.eventlogs

@@ -0,0 +1,12 @@
+{
+  "description" : "win.eventlogs",
+  "processors" : [ 
+    { "set":           { "if": "ctx.winlog?.channel != null",   "field": "event.module", "value": "windows_eventlog", "override": false, "ignore_failure": true }  },
+    { "set":           { "if": "ctx.winlog?.channel != null",   "field": "event.dataset", "value": "{{winlog.channel}}", "override": true }  },
+    { "set":           { "if": "ctx.winlog?.computer_name != null",   "field": "observer.name", "value": "{{winlog.computer_name}}", "override": true }  },
+    { "set":           { "field": "event.code", "value": "{{winlog.event_id}}", "override": true }  },  
+    { "set":           { "field": "event.category", "value": "host", "override": true }  },   
+    { "rename": 	   { "field": "winlog.event_data.SubjectUserName", 			  "target_field": "user.name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "winlog.event_data.User", 			            "target_field": "user.name",		"ignore_missing": true 	} }
+  ]
+}

salt/elasticsearch/files/ingest/zeek.intel

@@ -3,6 +3,7 @@
   "processors" : [
     { "remove":         { "field": ["host"],     "ignore_failure": true                                                                  } },
     { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
+    { "dot_expander":   { "field": "seen.indicator",    "path": "message2",                     "ignore_failure": true  } },
     { "rename": 	{ "field": "message2.seen.indicator", 	"target_field": "intel.indicator",		"ignore_missing": true 	} },
     { "dot_expander": 	{ "field": "seen.indicator_type", 	"path": "message2",			"ignore_failure": true 	} },
     { "rename": 	{ "field": "message2.seen.indicator_type", 	"target_field": "intel.indicator_type",		"ignore_missing": true 	} },

salt/elasticsearch/files/scripts/so-catrust

@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+# Copyright 2014,2015,2016,2017,2018,2019 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+{%- set VERSION = salt['pillar.get']('global:soversion', '') %}
+{%- set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
+{%- set MANAGER = salt['grains.get']('master') %}
+. /usr/sbin/so-common
+# Check to see if we have extracted the ca cert.
+if [ ! -f /opt/so/saltstack/local/salt/common/cacerts ]; then
+    docker run -v /etc/pki/ca.crt:/etc/pki/ca.crt --name so-elasticsearchca --user root --entrypoint jdk/bin/keytool {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elasticsearch:{{ VERSION }} -keystore /etc/pki/ca-trust/extracted/java/cacerts -alias SOSCA -import -file /etc/pki/ca.crt -storepass changeit -noprompt
+    docker cp so-elasticsearchca:/etc/pki/ca-trust/extracted/java/cacerts /opt/so/saltstack/local/salt/common/cacerts
+    docker cp so-elasticsearchca:/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /opt/so/saltstack/local/salt/common/tls-ca-bundle.pem
+    docker rm so-elasticsearchca
+    echo "" >> /opt/so/saltstack/local/salt/common/tls-ca-bundle.pem
+    echo "sosca" >> /opt/so/saltstack/local/salt/common/tls-ca-bundle.pem
+    cat /etc/pki/ca.crt >> /opt/so/saltstack/local/salt/common/tls-ca-bundle.pem
+else
+    exit 0
+fi

salt/elasticsearch/files/sotls.yml

@@ -0,0 +1,12 @@
+keystore.path: /usr/share/elasticsearch/config/sokeys
+keystore.password: changeit 
+keystore.algorithm: SunX509
+truststore.path: /etc/pki/java/cacerts
+truststore.password: changeit
+truststore.algorithm: PKIX
+protocols:
+- TLSv1.2
+ciphers:
+- TLS_RSA_WITH_AES_128_CBC_SHA256
+transport.encrypted: true
+http.encrypted: false

salt/elasticsearch/init.sls

@@ -12,23 +12,32 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'elasticsearch' in top_states %}
+
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
 {% set FEATURES = salt['pillar.get']('elastic:features', False) %}
+{%- set NODEIP = salt['pillar.get']('elasticsearch:mainip', '') -%}
 
-{% if FEATURES %}
-  {% set FEATURES = "-features" %}
+
+{%- if FEATURES is sameas true %}
+  {% set FEATUREZ = "-features" %}
 {% else %}
-  {% set FEATURES = '' %}
+  {% set FEATUREZ = '' %}
 {% endif %}
 
-{% if grains['role'] in ['so-eval','so-managersearch', 'so-manager', 'so-standalone'] %}
+{% if grains['role'] in ['so-eval','so-managersearch', 'so-manager', 'so-standalone', 'so-import'] %}
   {% set esclustername = salt['pillar.get']('manager:esclustername', '') %}
   {% set esheap = salt['pillar.get']('manager:esheap', '') %}
+  {% set ismanager = True %}
 {% elif grains['role'] in ['so-node','so-heavynode'] %}
   {% set esclustername = salt['pillar.get']('elasticsearch:esclustername', '') %}
   {% set esheap = salt['pillar.get']('elasticsearch:esheap', '') %}
+  {% set ismanager = False %}
 {% endif %}
 
 {% set TEMPLATES = salt['pillar.get']('elasticsearch:templates', {}) %}
@@ -37,6 +46,46 @@ vm.max_map_count:
   sysctl.present:
     - value: 262144
 
+{% if ismanager %}
+# We have to add the Manager CA to the CA list
+cascriptsync:
+  file.managed:
+    - name: /usr/sbin/so-catrust
+    - source: salt://elasticsearch/files/scripts/so-catrust
+    - user: 939
+    - group: 939
+    - mode: 750
+    - template: jinja
+
+# Run the CA magic
+cascriptfun:
+  cmd.run:
+    - name: /usr/sbin/so-catrust
+
+{% endif %}
+
+# Move our new CA over so Elastic and Logstash can use SSL with the internal CA
+catrustdir:
+  file.directory:
+    - name: /opt/so/conf/ca
+    - user: 939
+    - group: 939
+    - makedirs: True
+
+cacertz:
+  file.managed:
+    - name: /opt/so/conf/ca/cacerts
+    - source: salt://common/cacerts
+    - user: 939
+    - group: 939
+
+capemz:
+  file.managed:
+    - name: /opt/so/conf/ca/tls-ca-bundle.pem
+    - source: salt://common/tls-ca-bundle.pem
+    - user: 939
+    - group: 939
+
 # Add ES Group
 elasticsearchgroup:
   group.present:
@@ -95,6 +144,13 @@ esyml:
     - group: 939
     - template: jinja
 
+sotls:
+  file.managed:
+    - name: /opt/so/conf/elasticsearch/sotls.yml
+    - source: salt://elasticsearch/files/sotls.yml
+    - user: 930
+    - group: 939
+
 #sync templates to /opt/so/conf/elasticsearch/templates
 {% for TEMPLATE in TEMPLATES %}
 es_template_{{TEMPLATE.split('.')[0] | replace("/","_") }}:
@@ -126,18 +182,23 @@ eslogdir:
 
 so-elasticsearch:
   docker_container.running:
-    - image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elasticsearch:{{ VERSION }}{{ FEATURES }}
+    - image: {{ MANAGER }}:5000/{{ IMAGEREPO }}/so-elasticsearch:{{ VERSION }}{{ FEATUREZ }}
     - hostname: elasticsearch
     - name: so-elasticsearch
     - user: elasticsearch
+    - extra_hosts: 
+      - {{ grains.host }}:{{ NODEIP }}
+      {%- if ismanager %}
+      {%- if salt['pillar.get']('nodestab', {}) %}
+      {%- for SN, SNDATA in salt['pillar.get']('nodestab', {}).items() %}
+      - {{ SN.split('_')|first }}:{{ SNDATA.ip }}
+      {%- endfor %}
+      {%- endif %}
+      {%- endif %}
     - environment:
       - discovery.type=single-node
-      #- bootstrap.memory_lock=true
-      #- cluster.name={{ esclustername }}
       - ES_JAVA_OPTS=-Xms{{ esheap }} -Xmx{{ esheap }}
-      #- http.host=0.0.0.0
-      #- transport.host=127.0.0.1
-    - ulimits:
+      ulimits:
       - memlock=-1:-1
       - nofile=65536:65536
       - nproc=4096
@@ -149,6 +210,16 @@ so-elasticsearch:
       - /opt/so/conf/elasticsearch/log4j2.properties:/usr/share/elasticsearch/config/log4j2.properties:ro
       - /nsm/elasticsearch:/usr/share/elasticsearch/data:rw
       - /opt/so/log/elasticsearch:/var/log/elasticsearch:rw
+      - /opt/so/conf/ca/cacerts:/etc/pki/ca-trust/extracted/java/cacerts:ro
+      - /etc/pki/ca.crt:/usr/share/elasticsearch/config/ca.crt:ro
+      - /etc/pki/elasticsearch.p12:/usr/share/elasticsearch/config/elasticsearch.p12:ro
+      - /opt/so/conf/elasticsearch/sotls.yml:/usr/share/elasticsearch/config/sotls.yml:ro
+
+    - watch:
+      - file: cacertz
+      - file: esyml
+      - file: esingestconf
+      - file: so-elasticsearch-pipelines-file
 
 so-elasticsearch-pipelines-file:
   file.managed:
@@ -166,9 +237,17 @@ so-elasticsearch-pipelines:
       - file: esyml
       - file: so-elasticsearch-pipelines-file
 
-{% if grains['role'] in ['so-manager', 'so-eval', 'so-managersearch', 'so-standalone'] and TEMPLATES %}
+{% if grains['role'] in ['so-manager', 'so-eval', 'so-managersearch', 'so-standalone', 'so-heavynode', 'so-searchnode', 'so-import'] and TEMPLATES %}
 so-elasticsearch-templates:
   cmd.run:
     - name: /usr/sbin/so-elasticsearch-templates
     - cwd: /opt/so
-{% endif %}
+{% endif %}
+
+{% else %}
+
+elasticsearch_state_not_allowed:
+  test.fail_without_changes:
+    - name: elasticsearch_state_not_allowed
+
+{% endif %}

salt/elasticsearch/templates/so/so-common-template.json

@@ -6,11 +6,61 @@
     "number_of_replicas":0,
     "number_of_shards":1,
     "index.refresh_interval":"30s",
-    "index.routing.allocation.require.box_type":"hot"
+    "index.routing.allocation.require.box_type":"hot",
+    "analysis": {
+      "analyzer": {
+        "es_security_analyzer": {
+          "type": "custom",
+          "filter": [ "path_hierarchy_pattern_filter", "lowercase" ],
+          "tokenizer": "whitespace"
+        },
+        "es_security_search_analyzer": {
+          "type": "custom",
+          "filter": [ "lowercase" ],
+          "tokenizer": "whitespace"
+        },
+        "es_security_search_quote_analyzer": {
+          "type": "custom",
+          "filter": [ "lowercase" ],
+          "tokenizer": "whitespace"
+        }
+      },
+      "filter" : {
+        "path_hierarchy_pattern_filter": {
+          "type" : "pattern_capture",
+          "preserve_original": true,
+          "patterns": [
+            "((?:[^\\\\]*\\\\)*)(.*)",
+            "((?:[^/]*/)*)(.*)"
+          ]
+        }
+      }
+    }
   },
   "mappings":{
        "dynamic":false,
        "date_detection":false,
+       "dynamic_templates": [
+        {
+          "strings": {
+            "match_mapping_type": "string",
+            "mapping": {
+              "type": "text",
+              "fields": {
+                "keyword": {
+                 "type": "keyword"
+               },
+               "security": {
+                 "type": "text",
+                 "analyzer": "es_security_analyzer", 
+                 "search_analyzer": "es_security_search_analyzer", 
+                 "search_quote_analyzer": "es_security_search_quote_analyzer" 
+               }
+             }
+            }
+          }
+        }
+      ],
        "properties":{
         "@timestamp":{
           "type":"date"
@@ -18,7 +68,7 @@
         "@version":{
           "type":"keyword"
         },
-  "osquery":{
+        "osquery":{
           "type":"object",
           "dynamic":true
         },
@@ -85,7 +135,7 @@
           "type":"object",
           "dynamic": true
         },
-	"client":{
+	      "client":{
           "type":"object",
           "dynamic": true
         },
@@ -177,6 +227,10 @@
           "type":"object",
           "dynamic": true
         },
+        "import":{
+          "type":"object",
+          "dynamic": true
+        },
         "ingest":{
           "type":"object",
           "dynamic": true
@@ -185,7 +239,7 @@
           "type":"object",
           "dynamic": true
         },
-	"irc":{
+	      "irc":{
           "type":"object",
           "dynamic": true
         },
@@ -201,7 +255,7 @@
           "type":"object",
           "dynamic": true
         },
-	"message":{  
+	      "message":{  
           "type":"text",
           "fields":{  
             "keyword":{  
@@ -213,7 +267,7 @@
           "type":"object",
           "dynamic": true
         },
-	"mysql":{
+	      "mysql":{
           "type":"object",
           "dynamic": true
         },
@@ -221,7 +275,7 @@
           "type":"object",
           "dynamic": true
         },
-	"notice":{
+	      "notice":{
           "type":"object",
           "dynamic": true
         },
@@ -269,7 +323,7 @@
           "type":"object",
           "dynamic": true
         },
-	"request":{
+	      "request":{
           "type":"object",
           "dynamic": true
         },
@@ -281,7 +335,7 @@
           "type":"object",
           "dynamic": true
         },
-	"scan":{
+	      "scan":{
           "type":"object",
           "dynamic": true
         },
@@ -317,7 +371,7 @@
           "type":"object",
           "dynamic": true
         },
-	"source":{
+	      "source":{
           "type":"object",
           "dynamic": true
         },
@@ -329,7 +383,7 @@
           "type":"object",
           "dynamic": true
         },
-	"syslog":{
+	      "syslog":{
           "type":"object",
           "dynamic": true
         },
@@ -383,8 +437,16 @@
         },
         "winlog":{
           "type":"object",
-          "dynamic": true
-        },
+          "dynamic": true,
+          "properties":{
+            "event_id":{
+              "type":"long"
+            },
+            "event_data":{
+              "type":"object"
+            }
+        }
+      },
         "x509":{
           "type":"object",
           "dynamic": true

salt/filebeat/etc/filebeat.yml

@@ -1,16 +1,16 @@
 {%- if grains.role == 'so-heavynode' %}
-{%- set MANAGER = salt['pillar.get']('sensor:mainip' '') %}
+{%- set MANAGER = salt['grains.get']('host' '') %}
 {%- else %}
 {%- set MANAGER = salt['grains.get']('master') %}
 {%- endif %}
 
 
 {%- set HOSTNAME = salt['grains.get']('host', '') %}
-{%- set ZEEKVER = salt['pillar.get']('static:zeekversion', 'COMMUNITY') %}
-{%- set WAZUHENABLED = salt['pillar.get']('static:wazuh', '0') %}
+{%- set ZEEKVER = salt['pillar.get']('global:mdengine', 'COMMUNITY') %}
+{%- set WAZUHENABLED = salt['pillar.get']('global:wazuh', '0') %}
 {%- set STRELKAENABLED = salt['pillar.get']('strelka:enabled', '0') %}
-{%- set FLEETMANAGER = salt['pillar.get']('static:fleet_manager', False) -%}
-{%- set FLEETNODE = salt['pillar.get']('static:fleet_node', False) -%}
+{%- set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
+{%- set FLEETNODE = salt['pillar.get']('global:fleet_node', False) -%}
 
 name: {{ HOSTNAME }}
 
@@ -74,7 +74,7 @@ filebeat.modules:
 # List of prospectors to fetch data.
 filebeat.inputs:
 #------------------------------ Log prospector --------------------------------
-{%- if grains['role'] == 'so-sensor' or grains['role'] == "so-eval" or grains['role'] == "so-helix" or grains['role'] == "so-heavynode" or grains['role'] == "so-standalone" %}
+{%- if grains['role'] in ['so-sensor', "so-eval", "so-helix", "so-heavynode", "so-standalone", "so-import"] %}
 - type: udp
   enabled: true
   host: "0.0.0.0:514"
@@ -253,7 +253,7 @@ output.{{ type }}:
   {%- endfor %}
 {%- else %}
 #----------------------------- Elasticsearch/Logstash output ---------------------------------
-  {%- if grains['role'] == "so-eval" %}
+  {%- if grains['role'] in ["so-eval", "so-import"] %}
 output.elasticsearch:
   enabled: true
   hosts: ["{{ MANAGER }}:9200"]

salt/filebeat/init.sls

@@ -11,12 +11,17 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'filebeat' in top_states %}
+
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
-{% set MANAGERIP = salt['pillar.get']('static:managerip', '') %}
+{% set MANAGERIP = salt['pillar.get']('global:managerip', '') %}
 {% set FEATURES = salt['pillar.get']('elastic:features', False) %}
-{% if FEATURES %}
+{%- if FEATURES is sameas true %}
   {% set FEATURES = "-features" %}
 {% else %}
   {% set FEATURES = '' %}
@@ -60,8 +65,8 @@ so-filebeat:
       - /nsm:/nsm:ro
       - /opt/so/log/filebeat:/usr/share/filebeat/logs:rw
       - /opt/so/conf/filebeat/etc/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
-      - /opt/so/wazuh/logs/alerts:/wazuh/alerts:ro
-      - /opt/so/wazuh/logs/archives:/wazuh/archives:ro
+      - /nsm/wazuh/logs/alerts:/wazuh/alerts:ro
+      - /nsm/wazuh/logs/archives:/wazuh/archives:ro
       - /opt/so/conf/filebeat/etc/pki/filebeat.crt:/usr/share/filebeat/filebeat.crt:ro
       - /opt/so/conf/filebeat/etc/pki/filebeat.key:/usr/share/filebeat/filebeat.key:ro
       - /etc/ssl/certs/intca.crt:/usr/share/filebeat/intraca.crt:ro
@@ -69,3 +74,11 @@ so-filebeat:
         - 0.0.0.0:514:514/udp
     - watch:
       - file: /opt/so/conf/filebeat/etc/filebeat.yml
+
+{% else %}
+
+filebeat_state_not_allowed:
+  test.fail_without_changes:
+    - name: filebeat_state_not_allowed
+
+{% endif %}

salt/firewall/assigned_hostgroups.map.yaml

@@ -1,3 +1,4 @@
+{% set ISAIRGAP = salt['pillar.get']('global:airgap', 'False') %}
 {% import_yaml 'firewall/portgroups.yaml' as portgroups %}
 {% set portgroups = portgroups.firewall.aliases.ports %}
 
@@ -15,6 +16,7 @@ role:
               - {{ portgroups.mysql }}
               - {{ portgroups.kibana }}
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.influxdb }}
               - {{ portgroups.fleet_api }}
               - {{ portgroups.cortex }}
@@ -38,6 +40,7 @@ role:
           search_node:
             portgroups:
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.elasticsearch_node }}
           self:
             portgroups:
@@ -99,6 +102,7 @@ role:
               - {{ portgroups.mysql }}
               - {{ portgroups.kibana }}
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.influxdb }}
               - {{ portgroups.fleet_api }}
               - {{ portgroups.cortex }}
@@ -106,6 +110,9 @@ role:
               - {{ portgroups.elasticsearch_node }}
               - {{ portgroups.cortex_es_rest }}
               - {{ portgroups.cortex_es_node }}
+              {% if ISAIRGAP is sameas true %}
+              - {{ portgroups.agrules }}
+              {% endif %}
           minion:
             portgroups:
               - {{ portgroups.acng }}
@@ -114,6 +121,9 @@ role:
               - {{ portgroups.influxdb }}
               - {{ portgroups.wazuh_api }}
               - {{ portgroups.fleet_api }}
+              {% if ISAIRGAP is sameas true %}
+              - {{ portgroups.yum }}
+              {% endif %}
           sensor:
             portgroups:
               - {{ portgroups.sensoroni }}
@@ -122,6 +132,7 @@ role:
           search_node:
             portgroups:
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.elasticsearch_node }}
           self:
             portgroups:
@@ -180,6 +191,7 @@ role:
               - {{ portgroups.mysql }}
               - {{ portgroups.kibana }}
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.influxdb }}
               - {{ portgroups.fleet_api }}
               - {{ portgroups.cortex }}
@@ -195,6 +207,7 @@ role:
               - {{ portgroups.influxdb }}
               - {{ portgroups.wazuh_api }}
               - {{ portgroups.fleet_api }}
+              - {{ portgroups.yum }}
           sensor:
             portgroups:
               - {{ portgroups.sensoroni }}
@@ -203,6 +216,7 @@ role:
           search_node:
             portgroups:
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.elasticsearch_node }}
           self:
             portgroups:
@@ -261,6 +275,7 @@ role:
               - {{ portgroups.mysql }}
               - {{ portgroups.kibana }}
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.influxdb }}
               - {{ portgroups.fleet_api }}
               - {{ portgroups.cortex }}
@@ -275,7 +290,8 @@ role:
               - {{ portgroups.osquery_8080 }}
               - {{ portgroups.influxdb }}
               - {{ portgroups.wazuh_api }}
-              - {{ portgroups.fleet_api }}              
+              - {{ portgroups.fleet_api }}
+              - {{ portgroups.yum }}              
           sensor:
             portgroups:
               - {{ portgroups.sensoroni }}
@@ -284,6 +300,7 @@ role:
           search_node:
             portgroups:
               - {{ portgroups.redis }}
+              - {{ portgroups.minio }}
               - {{ portgroups.elasticsearch_node }}
           self:
             portgroups:
@@ -434,16 +451,24 @@ role:
     chain:
       DOCKER-USER:
         hostgroups:
-          self:
+          manager:
             portgroups:
-              - {{ portgroups.redis }}
-              - {{ portgroups.beats_5044 }}
-              - {{ portgroups.beats_5644 }}
+              - {{ portgroups.elasticsearch_node }}
+          dockernet:
+            portgroups:
+              - {{ portgroups.elasticsearch_node }}
+              - {{ portgroups.elasticsearch_rest }}
+          elasticsearch_rest:
+            portgroups:
+              - {{ portgroups.elasticsearch_rest }}
       INPUT:
         hostgroups:
           anywhere:
             portgroups:
               - {{ portgroups.ssh }}
+          dockernet:
+            portgroups:
+              - {{ portgroups.all }}
           localhost:
             portgroups:
               - {{ portgroups.all }}
@@ -480,3 +505,55 @@ role:
           localhost:
             portgroups:
               - {{ portgroups.all }}
+  import:
+    chain:
+      DOCKER-USER:
+        hostgroups:
+          manager:
+            portgroups:
+              - {{ portgroups.kibana }}
+              - {{ portgroups.redis }}
+              - {{ portgroups.influxdb }}
+              - {{ portgroups.elasticsearch_rest }}
+              - {{ portgroups.elasticsearch_node }}
+          minion:
+            portgroups:
+              - {{ portgroups.docker_registry }}
+          sensor:
+            portgroups:
+              - {{ portgroups.beats_5044 }}
+              - {{ portgroups.beats_5644 }}
+              - {{ portgroups.sensoroni }}
+          search_node:
+            portgroups:
+              - {{ portgroups.redis }}
+              - {{ portgroups.elasticsearch_node }}
+          self:
+            portgroups:
+              - {{ portgroups.syslog}}
+          beats_endpoint:
+            portgroups:
+              - {{ portgroups.beats_5044 }}
+          beats_endpoint_ssl:
+            portgroups:
+              - {{ portgroups.beats_5644 }}
+          elasticsearch_rest:
+            portgroups:
+              - {{ portgroups.elasticsearch_rest }}
+          analyst:
+            portgroups:
+              - {{ portgroups.nginx }}
+      INPUT:
+        hostgroups:
+          anywhere:
+            portgroups:
+              - {{ portgroups.ssh }}
+          dockernet:
+            portgroups:
+              - {{ portgroups.all }}
+          localhost:
+            portgroups:
+              - {{ portgroups.all }}
+          minion:
+            portgroups:
+              - {{ portgroups.salt_manager }}

salt/firewall/init.sls

@@ -1,3 +1,8 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'firewall' in top_states %}
+
 # Firewall Magic for the grid
 {% from 'firewall/map.jinja' import hostgroups with context %}
 {% from 'firewall/map.jinja' import assigned_hostgroups with context %}
@@ -128,3 +133,11 @@ iptables_drop_all_the_things:
     - chain: LOGGING
     - jump: DROP
     - save: True
+
+{% else %}
+
+firewall_state_not_allowed:
+  test.fail_without_changes:
+    - name: firewall_state_not_allowed
+
+{% endif %}

salt/firewall/portgroups.yaml

@@ -9,6 +9,9 @@ firewall:
       acng:
         tcp:
           - 3142
+      agrules:
+        tcp:
+          - 7788
       beats_5044:
         tcp:
           - 5044
@@ -45,6 +48,9 @@ firewall:
       kibana:
         tcp:
           - 5601
+      minio:
+        tcp:
+          - 9595
       mysql:
         tcp:
           - 3306
@@ -61,6 +67,7 @@ firewall:
       redis:
         tcp:
           - 6379
+          - 9696
       salt_manager:
         tcp:
           - 4505
@@ -90,3 +97,6 @@ firewall:
       wazuh_authd:
         tcp:
           - 1515
+      yum:
+        tcp:
+          - 443

salt/fleet/event_enable-fleet.sls

@@ -1,4 +1,4 @@
-{% set ENROLLSECRET = salt['cmd.run']('docker exec so-fleet fleetctl get enroll-secret') %}
+{% set ENROLLSECRET = salt['cmd.run']('docker exec so-fleet fleetctl get enroll-secret default') %}
 {% set MAININT = salt['pillar.get']('host:mainint') %}
 {% set MAINIP = salt['grains.get']('ip_interfaces').get(MAININT)[0] %}
 

salt/fleet/event_gen-packages.sls

@@ -1,17 +1,17 @@
 {% set MANAGER = salt['grains.get']('master') %}
 {% set ENROLLSECRET = salt['pillar.get']('secrets:fleet_enroll-secret') %}
-{% set CURRENTPACKAGEVERSION = salt['pillar.get']('static:fleet_packages-version') %}
-{% set VERSION = salt['pillar.get']('static:soversion') %}
-{% set CUSTOM_FLEET_HOSTNAME = salt['pillar.get']('static:fleet_custom_hostname', None) %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
-{%- set FLEETNODE = salt['pillar.get']('static:fleet_node') -%}
+{% set CURRENTPACKAGEVERSION = salt['pillar.get']('global:fleet_packages-version') %}
+{% set VERSION = salt['pillar.get']('global:soversion') %}
+{% set CUSTOM_FLEET_HOSTNAME = salt['pillar.get']('global:fleet_custom_hostname', None) %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
+{%- set FLEETNODE = salt['pillar.get']('global:fleet_node') -%}
 
 {% if CUSTOM_FLEET_HOSTNAME != None and CUSTOM_FLEET_HOSTNAME != '' %}
    {% set HOSTNAME =  CUSTOM_FLEET_HOSTNAME  %}
 {% elif FLEETNODE %}
    {% set HOSTNAME = grains.host  %}
 {% else %}
-   {% set HOSTNAME = salt['pillar.get']('manager:url_base')  %}
+   {% set HOSTNAME = salt['pillar.get']('global:url_base')  %}
 {% endif %}
 
 so/fleet:

salt/fleet/event_update-custom-hostname.sls

@@ -1,4 +1,4 @@
-{% set CUSTOM_FLEET_HOSTNAME = salt['pillar.get']('static:fleet_custom_hostname', None) %}
+{% set CUSTOM_FLEET_HOSTNAME = salt['pillar.get']('global:fleet_custom_hostname', None) %}
 
 so/fleet:
   event.send:

salt/fleet/files/packs/osquery-config.conf

@@ -22,6 +22,8 @@ spec:
       distributed_tls_max_attempts: 3
       distributed_tls_read_endpoint: /api/v1/osquery/distributed/read
       distributed_tls_write_endpoint: /api/v1/osquery/distributed/write
+      enable_windows_events_publisher: true
+      enable_windows_events_subscriber: true
       logger_plugin: tls
       logger_tls_endpoint: /api/v1/osquery/log
       logger_tls_period: 10

salt/fleet/init.sls

@@ -1,8 +1,8 @@
 {%- set MYSQLPASS = salt['pillar.get']('secrets:mysql', None) -%}
 {%- set FLEETPASS = salt['pillar.get']('secrets:fleet', None) -%}
 {%- set FLEETJWT = salt['pillar.get']('secrets:fleet_jwt', None) -%}
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
 {% set FLEETARCH = salt['grains.get']('role') %}
 
@@ -10,7 +10,7 @@
   {% set MAININT = salt['pillar.get']('host:mainint') %}
   {% set MAINIP = salt['grains.get']('ip_interfaces').get(MAININT)[0] %}
 {% else %}
-  {% set MAINIP = salt['pillar.get']('static:managerip') %}
+  {% set MAINIP = salt['pillar.get']('global:managerip') %}
 {% endif %}
 
 include:
@@ -132,4 +132,4 @@ so-fleet:
     - watch:
       - /opt/so/conf/fleet/etc
 
-{% endif %}
+{% endif %}

salt/fleet/install_package.sls

@@ -1,8 +1,8 @@
-{%- set FLEETMANAGER = salt['pillar.get']('static:fleet_manager', False) -%}
-{%- set FLEETNODE = salt['pillar.get']('static:fleet_node', False) -%}
-{%- set FLEETHOSTNAME = salt['pillar.get']('static:fleet_hostname', False) -%}
-{%- set FLEETIP = salt['pillar.get']('static:fleet_ip', False) -%}
-{% set CUSTOM_FLEET_HOSTNAME = salt['pillar.get']('static:fleet_custom_hostname', None) %}
+{%- set FLEETMANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
+{%- set FLEETNODE = salt['pillar.get']('global:fleet_node', False) -%}
+{%- set FLEETHOSTNAME = salt['pillar.get']('global:fleet_hostname', False) -%}
+{%- set FLEETIP = salt['pillar.get']('global:fleet_ip', False) -%}
+{% set CUSTOM_FLEET_HOSTNAME = salt['pillar.get']('global:fleet_custom_hostname', None) %}
 
 {% if CUSTOM_FLEET_HOSTNAME != (None and '') %}
 

salt/freqserver/init.sls

@@ -12,8 +12,12 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
 
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% if 'freqserver' in top_states %}
+
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 
 # Create the user
 fservergroup:
@@ -52,3 +56,11 @@ so-freq:
     - binds:
       - /opt/so/log/freq_server:/var/log/freq_server:rw
 
+{% else %}
+
+freqserver_state_not_allowed:
+  test.fail_without_changes:
+    - name: freqserver_state_not_allowed
+
+{% endif %}
+

salt/grafana/dashboards/eval/eval.json

@@ -3860,70 +3860,6 @@
                 "value": "{{ MONINT }}"
               }
             ]
-          },
-          {
-            "alias": "Outbound",
-            "dsType": "influxdb",
-            "groupBy": [
-              {
-                "params": [
-                  "$Interval"
-                ],
-                "type": "time"
-              },
-              {
-                "params": [
-                  "null"
-                ],
-                "type": "fill"
-              }
-            ],
-            "measurement": "net",
-            "orderByTime": "ASC",
-            "policy": "default",
-            "query": "SELECT 8 * derivative(mean(\"bytes_sent\"),1s) FROM \"net\" WHERE \"host\" = 'JumpHost' AND \"interface\" = 'eth0' AND $timeFilter GROUP BY time($interval) fill(null)",
-            "rawQuery": false,
-            "refId": "B",
-            "resultFormat": "time_series",
-            "select": [
-              [
-                {
-                  "params": [
-                    "bytes_sent"
-                  ],
-                  "type": "field"
-                },
-                {
-                  "params": [],
-                  "type": "mean"
-                },
-                {
-                  "params": [
-                    "1s"
-                  ],
-                  "type": "derivative"
-                },
-                {
-                  "params": [
-                    "*8"
-                  ],
-                  "type": "math"
-                }
-              ]
-            ],
-            "tags": [
-              {
-                "key": "host",
-                "operator": "=",
-                "value": "{{ SERVERNAME }}"
-              },
-              {
-                "condition": "AND",
-                "key": "interface",
-                "operator": "=",
-                "value": "{{ MONINT }}"
-              }
-            ]
           }
         ],
         "thresholds": [],

salt/grafana/dashboards/manager/manager.json

@@ -583,7 +583,7 @@
       "timeFrom": null,
       "timeRegions": [],
       "timeShift": null,
-      "title": "{{ SERVERNAME }} - REDIS Unparsed Queue",
+      "title": "{{ SERVERNAME }} - Redis Queue",
       "tooltip": {
         "shared": true,
         "sort": 0,
@@ -621,134 +621,6 @@
         "alignLevel": null
       }
     },
-    {
-      "aliasColors": {},
-      "bars": false,
-      "cacheTimeout": null,
-      "dashLength": 10,
-      "dashes": false,
-      "datasource": "InfluxDB",
-      "fill": 1,
-      "fillGradient": 0,
-      "gridPos": {
-        "h": 5,
-        "w": 4,
-        "x": 20,
-        "y": 0
-      },
-      "hiddenSeries": false,
-      "id": 21,
-      "legend": {
-        "avg": false,
-        "current": false,
-        "max": false,
-        "min": false,
-        "show": false,
-        "total": false,
-        "values": false
-      },
-      "lines": true,
-      "linewidth": 1,
-      "links": [],
-      "nullPointMode": "connected",
-      "options": {
-        "dataLinks": []
-      },
-      "percentage": false,
-      "pointradius": 2,
-      "points": false,
-      "renderer": "flot",
-      "seriesOverrides": [],
-      "spaceLength": 10,
-      "stack": false,
-      "steppedLine": false,
-      "targets": [
-        {
-          "dsType": "influxdb",
-          "groupBy": [
-            {
-              "params": [
-                "$Interval"
-              ],
-              "type": "time"
-            },
-            {
-              "params": [
-                "null"
-              ],
-              "type": "fill"
-            }
-          ],
-          "measurement": "redisqueue",
-          "orderByTime": "ASC",
-          "policy": "default",
-          "refId": "A",
-          "resultFormat": "time_series",
-          "select": [
-            [
-              {
-                "params": [
-                  "parsed"
-                ],
-                "type": "field"
-              },
-              {
-                "params": [],
-                "type": "mean"
-              }
-            ]
-          ],
-          "tags": [
-            {
-              "key": "host",
-              "operator": "=",
-              "value": "{{ SERVERNAME }}"
-            }
-          ]
-        }
-      ],
-      "thresholds": [],
-      "timeFrom": null,
-      "timeRegions": [],
-      "timeShift": null,
-      "title": "{{ SERVERNAME }} - REDIS Parsed Queue",
-      "tooltip": {
-        "shared": true,
-        "sort": 0,
-        "value_type": "individual"
-      },
-      "type": "graph",
-      "xaxis": {
-        "buckets": null,
-        "mode": "time",
-        "name": null,
-        "show": true,
-        "values": []
-      },
-      "yaxes": [
-        {
-          "decimals": 0,
-          "format": "short",
-          "label": null,
-          "logBase": 1,
-          "max": null,
-          "min": null,
-          "show": true
-        },
-        {
-          "format": "short",
-          "label": null,
-          "logBase": 1,
-          "max": null,
-          "min": null,
-          "show": false
-        }
-      ],
-      "yaxis": {
-        "align": false,
-        "alignLevel": null
-      }
-    },
     {
       "cacheTimeout": null,
       "datasource": "InfluxDB",
@@ -1351,7 +1223,7 @@
       "timeFrom": null,
       "timeRegions": [],
       "timeShift": null,
-      "title": "{{ SERVERNAME }} - REDIS CPU Usage",
+      "title": "{{ SERVERNAME }} - Redis CPU Usage",
       "tooltip": {
         "shared": true,
         "sort": 0,
@@ -1485,7 +1357,7 @@
       "timeFrom": null,
       "timeRegions": [],
       "timeShift": null,
-      "title": "{{ SERVERNAME }} - REDIS Memory Usage",
+      "title": "{{ SERVERNAME }} - Redis Memory Usage",
       "tooltip": {
         "shared": true,
         "sort": 0,
@@ -4043,6 +3915,138 @@
         "align": false,
         "alignLevel": null
       }
+    },
+    {
+      "aliasColors": {},
+      "bars": false,
+      "dashLength": 10,
+      "dashes": false,
+      "datasource": "InfluxDB",
+      "fieldConfig": {
+        "defaults": {
+          "custom": {}
+        },
+        "overrides": []
+      },
+      "fill": 1,
+      "fillGradient": 0,
+      "gridPos": {
+        "h": 5,
+        "w": 4,
+        "x": 20,
+        "y": 5
+      },
+      "hiddenSeries": false,
+      "id": 40,
+      "legend": {
+        "avg": false,
+        "current": false,
+        "max": false,
+        "min": false,
+        "show": false,
+        "total": false,
+        "values": false
+      },
+      "lines": true,
+      "linewidth": 1,
+      "nullPointMode": "connected",
+      "options": {
+        "dataLinks": []
+      },
+      "percentage": false,
+      "pointradius": 2,
+      "points": false,
+      "renderer": "flot",
+      "seriesOverrides": [],
+      "spaceLength": 10,
+      "stack": false,
+      "steppedLine": false,
+      "targets": [
+        {
+          "groupBy": [
+            {
+              "params": [
+                "$__interval"
+              ],
+              "type": "time"
+            },
+            {
+              "params": [
+                "null"
+              ],
+              "type": "fill"
+            }
+          ],
+          "measurement": "influxsize",
+          "orderByTime": "ASC",
+          "policy": "autogen",
+          "refId": "A",
+          "resultFormat": "time_series",
+          "select": [
+            [
+              {
+                "params": [
+                  "kbytes"
+                ],
+                "type": "field"
+              },
+              {
+                "params": [],
+                "type": "mean"
+              }
+            ]
+          ],
+          "tags": [
+            {
+              "key": "host",
+              "operator": "=",
+              "value": "{{ SERVERNAME }}"
+            }
+          ]
+        }
+      ],
+      "thresholds": [],
+      "timeFrom": null,
+      "timeRegions": [],
+      "timeShift": null,
+      "title": "{{ SERVERNAME }} - InfluxDB Size",
+      "tooltip": {
+        "shared": true,
+        "sort": 0,
+        "value_type": "individual"
+      },
+      "type": "graph",
+      "xaxis": {
+        "buckets": null,
+        "mode": "time",
+        "name": null,
+        "show": true,
+        "values": []
+      },
+      "yaxes": [
+        {
+          "$$hashKey": "object:526",
+          "format": "deckbytes",
+          "label": null,
+          "logBase": 1,
+          "max": null,
+          "min": null,
+          "show": true
+        },
+        {
+          "$$hashKey": "object:527",
+          "format": "short",
+          "label": null,
+          "logBase": 1,
+          "max": null,
+          "min": null,
+          "show": false
+        }
+      ],
+      "yaxis": {
+        "align": false,
+        "alignLevel": null
+      }
     }
   ],
   "refresh": false,

salt/grafana/dashboards/managersearch/managersearch.json

@@ -588,7 +588,7 @@
         "timeFrom": null,
         "timeRegions": [],
         "timeShift": null,
-        "title": "{{ SERVERNAME }} - REDIS Unparsed Queue",
+        "title": "{{ SERVERNAME }} - Redis Queue",
         "tooltip": {
           "shared": true,
           "sort": 0,
@@ -627,132 +627,6 @@
           "alignLevel": null
         }
       },
-      {
-        "aliasColors": {},
-        "bars": false,
-        "dashLength": 10,
-        "dashes": false,
-        "datasource": "InfluxDB",
-        "fill": 1,
-        "fillGradient": 0,
-        "gridPos": {
-          "h": 5,
-          "w": 4,
-          "x": 20,
-          "y": 0
-        },
-        "hiddenSeries": false,
-        "id": 51,
-        "legend": {
-          "avg": false,
-          "current": false,
-          "max": false,
-          "min": false,
-          "show": false,
-          "total": false,
-          "values": false
-        },
-        "lines": true,
-        "linewidth": 1,
-        "nullPointMode": "connected",
-        "options": {
-          "dataLinks": []
-        },
-        "percentage": false,
-        "pointradius": 2,
-        "points": false,
-        "renderer": "flot",
-        "seriesOverrides": [],
-        "spaceLength": 10,
-        "stack": false,
-        "steppedLine": false,
-        "targets": [
-          {
-            "groupBy": [
-              {
-                "params": [
-                  "$__interval"
-                ],
-                "type": "time"
-              },
-              {
-                "params": [
-                  "null"
-                ],
-                "type": "fill"
-              }
-            ],
-            "measurement": "redisqueue",
-            "orderByTime": "ASC",
-            "policy": "default",
-            "refId": "A",
-            "resultFormat": "time_series",
-            "select": [
-              [
-                {
-                  "params": [
-                    "parsed"
-                  ],
-                  "type": "field"
-                },
-                {
-                  "params": [],
-                  "type": "mean"
-                }
-              ]
-            ],
-            "tags": [
-              {
-                "key": "host",
-                "operator": "=",
-                "value": "{{ SERVERNAME }}"
-              }
-            ]
-          }
-        ],
-        "thresholds": [],
-        "timeFrom": null,
-        "timeRegions": [],
-        "timeShift": null,
-        "title": "{{ SERVERNAME }} - REDIS Parsed Queue",
-        "tooltip": {
-          "shared": true,
-          "sort": 0,
-          "value_type": "individual"
-        },
-        "type": "graph",
-        "xaxis": {
-          "buckets": null,
-          "mode": "time",
-          "name": null,
-          "show": true,
-          "values": []
-        },
-        "yaxes": [
-          {
-            "$$hashKey": "object:1367",
-            "format": "short",
-            "label": null,
-            "logBase": 1,
-            "max": null,
-            "min": null,
-            "show": true
-          },
-          {
-            "$$hashKey": "object:1368",
-            "format": "short",
-            "label": null,
-            "logBase": 1,
-            "max": null,
-            "min": null,
-            "show": false
-          }
-        ],
-        "yaxis": {
-          "align": false,
-          "alignLevel": null
-        }
-      },
       {
         "cacheTimeout": null,
         "datasource": "InfluxDB",
@@ -1352,7 +1226,7 @@
         "timeFrom": null,
         "timeRegions": [],
         "timeShift": null,
-        "title": "{{ SERVERNAME }} - REDIS CPU Usage",
+        "title": "{{ SERVERNAME }} - Redis CPU Usage",
         "tooltip": {
           "shared": true,
           "sort": 0,
@@ -1485,7 +1359,7 @@
         "timeFrom": null,
         "timeRegions": [],
         "timeShift": null,
-        "title": "{{ SERVERNAME }} - REDIS Memory Usage",
+        "title": "{{ SERVERNAME }} - Redis Memory Usage",
         "tooltip": {
           "shared": true,
           "sort": 0,
@@ -4787,6 +4661,138 @@
           "align": false,
           "alignLevel": null
         }
+      },
+      {
+        "aliasColors": {},
+        "bars": false,
+        "dashLength": 10,
+        "dashes": false,
+        "datasource": "InfluxDB",
+        "fieldConfig": {
+          "defaults": {
+            "custom": {}
+          },
+          "overrides": []
+        },
+        "fill": 1,
+        "fillGradient": 0,
+        "gridPos": {
+          "h": 5,
+          "w": 4,
+          "x": 20,
+          "y": 5
+        },
+        "hiddenSeries": false,
+        "id": 57,
+        "legend": {
+          "avg": false,
+          "current": false,
+          "max": false,
+          "min": false,
+          "show": false,
+          "total": false,
+          "values": false
+        },
+        "lines": true,
+        "linewidth": 1,
+        "nullPointMode": "connected",
+        "options": {
+          "dataLinks": []
+        },
+        "percentage": false,
+        "pointradius": 2,
+        "points": false,
+        "renderer": "flot",
+        "seriesOverrides": [],
+        "spaceLength": 10,
+        "stack": false,
+        "steppedLine": false,
+        "targets": [
+          {
+            "groupBy": [
+              {
+                "params": [
+                  "$__interval"
+                ],
+                "type": "time"
+              },
+              {
+                "params": [
+                  "null"
+                ],
+                "type": "fill"
+              }
+            ],
+            "measurement": "influxsize",
+            "orderByTime": "ASC",
+            "policy": "autogen",
+            "refId": "A",
+            "resultFormat": "time_series",
+            "select": [
+              [
+                {
+                  "params": [
+                    "kbytes"
+                  ],
+                  "type": "field"
+                },
+                {
+                  "params": [],
+                  "type": "mean"
+                }
+              ]
+            ],
+            "tags": [
+              {
+                "key": "host",
+                "operator": "=",
+                "value": "{{ SERVERNAME }}"
+              }
+            ]
+          }
+        ],
+        "thresholds": [],
+        "timeFrom": null,
+        "timeRegions": [],
+        "timeShift": null,
+        "title": "{{ SERVERNAME }} - InfluxDB Size",
+        "tooltip": {
+          "shared": true,
+          "sort": 0,
+          "value_type": "individual"
+        },
+        "type": "graph",
+        "xaxis": {
+          "buckets": null,
+          "mode": "time",
+          "name": null,
+          "show": true,
+          "values": []
+        },
+        "yaxes": [
+          {
+            "$$hashKey": "object:140",
+            "format": "deckbytes",
+            "label": null,
+            "logBase": 1,
+            "max": null,
+            "min": null,
+            "show": true
+          },
+          {
+            "$$hashKey": "object:141",
+            "format": "short",
+            "label": null,
+            "logBase": 1,
+            "max": null,
+            "min": null,
+            "show": false
+          }
+        ],
+        "yaxis": {
+          "align": false,
+          "alignLevel": null
+        }
       }
     ],
     "refresh": false,

salt/grafana/dashboards/sensor_nodes/sensor.json

@@ -3420,70 +3420,6 @@
               "value": "{{ MONINT }}"
             }
           ]
-        },
-        {
-          "alias": "OutBound",
-          "dsType": "influxdb",
-          "groupBy": [
-            {
-              "params": [
-                "$Interval"
-              ],
-              "type": "time"
-            },
-            {
-              "params": [
-                "null"
-              ],
-              "type": "fill"
-            }
-          ],
-          "measurement": "net",
-          "orderByTime": "ASC",
-          "policy": "default",
-          "query": "SELECT 8 * derivative(mean(\"bytes_sent\"),1s) FROM \"net\" WHERE \"host\" = 'JumpHost' AND \"interface\" = 'eth0' AND $timeFilter GROUP BY time($interval) fill(null)",
-          "rawQuery": false,
-          "refId": "B",
-          "resultFormat": "time_series",
-          "select": [
-            [
-              {
-                "params": [
-                  "bytes_sent"
-                ],
-                "type": "field"
-              },
-              {
-                "params": [],
-                "type": "mean"
-              },
-              {
-                "params": [
-                  "1s"
-                ],
-                "type": "derivative"
-              },
-              {
-                "params": [
-                  "*8"
-                ],
-                "type": "math"
-              }
-            ]
-          ],
-          "tags": [
-            {
-              "key": "host",
-              "operator": "=",
-              "value": "{{ SERVERNAME }}"
-            },
-            {
-              "condition": "AND",
-              "key": "interface",
-              "operator": "=",
-              "value": "{{ MONINT }}"
-            }
-          ]
         }
       ],
       "thresholds": [],

salt/grafana/etc/datasources/influxdb.yaml

@@ -1,4 +1,4 @@
-{%- set MANAGER = salt['pillar.get']('static:managerip', '') %}
+{%- set MANAGER = salt['pillar.get']('global:managerip', '') %}
 apiVersion: 1
 
 deleteDatasources:

salt/grafana/init.sls

@@ -1,7 +1,12 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'grafana' in top_states %}
+
 {% set GRAFANA = salt['pillar.get']('manager:grafana', '0') %}
 {% set MANAGER = salt['grains.get']('master') %}
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 
 {% if grains['role'] in ['so-manager', 'so-managersearch', 'so-eval', 'so-standalone'] and GRAFANA == 1 %}
 
@@ -91,7 +96,6 @@ dashboard-manager:
     - defaults:
       SERVERNAME: {{ SN }}
       MANINT: {{ SNDATA.manint }}
-      MONINT: {{ SNDATA.manint }}
       CPUS: {{ SNDATA.totalcpus }}
       UID: so_overview
       ROOTFS: {{ SNDATA.rootfs }}
@@ -114,7 +118,6 @@ dashboard-managersearch:
     - defaults:
       SERVERNAME: {{ SN }}
       MANINT: {{ SNDATA.manint }}
-      MONINT: {{ SNDATA.manint }}
       CPUS: {{ SNDATA.totalcpus }}
       UID: so_overview
       ROOTFS: {{ SNDATA.rootfs }}
@@ -137,7 +140,7 @@ dashboard-standalone:
     - defaults:
       SERVERNAME: {{ SN }}
       MANINT: {{ SNDATA.manint }}
-      MONINT: {{ SNDATA.manint }}
+      MONINT: {{ SNDATA.monint }}
       CPUS: {{ SNDATA.totalcpus }}
       UID: so_overview
       ROOTFS: {{ SNDATA.rootfs }}
@@ -159,8 +162,8 @@ dashboard-{{ SN }}:
     - source: salt://grafana/dashboards/sensor_nodes/sensor.json
     - defaults:
       SERVERNAME: {{ SN }}
-      MONINT: {{ SNDATA.monint }}
       MANINT: {{ SNDATA.manint }}
+      MONINT: {{ SNDATA.monint }}
       CPUS: {{ SNDATA.totalcpus }}
       UID: {{ SNDATA.guid }}
       ROOTFS: {{ SNDATA.rootfs }}
@@ -183,7 +186,6 @@ dashboardsearch-{{ SN }}:
     - defaults:
       SERVERNAME: {{ SN }}
       MANINT: {{ SNDATA.manint }}
-      MONINT: {{ SNDATA.manint }}
       CPUS: {{ SNDATA.totalcpus }}
       UID: {{ SNDATA.guid }}
       ROOTFS: {{ SNDATA.rootfs }}
@@ -233,4 +235,12 @@ so-grafana:
     - watch:
       - file: /opt/so/conf/grafana/*
 
+{% endif %}
+
+{% else %}
+
+grafana_state_not_allowed:
+  test.fail_without_changes:
+    - name: grafana_state_not_allowed
+
 {% endif %}

salt/healthcheck/init.sls

@@ -1,3 +1,8 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'healthcheck' in top_states %}
+
 {% set CHECKS = salt['pillar.get']('healthcheck:checks', {}) %}
 {% set ENABLED = salt['pillar.get']('healthcheck:enabled', False) %}
 {% set SCHEDULE = salt['pillar.get']('healthcheck:schedule', 300) %}
@@ -23,3 +28,11 @@ healthcheck_schedule_{{ STATUS[0] }}:
 healthcheck_schedule_{{ STATUS[1] }}:
   schedule.{{ STATUS[1] }}:
     - name: healthcheck
+
+{% else %}
+
+healthcheck_state_not_allowed:
+  test.fail_without_changes:
+    - name: healthcheck_state_not_allowed
+
+{% endif %}

salt/idstools/etc/rulecat.conf

@@ -1,21 +1,32 @@
 {%- set URLS = salt['pillar.get']('idstools:config:urls') -%}
 {%- set RULESET = salt['pillar.get']('idstools:config:ruleset') -%}
 {%- set OINKCODE = salt['pillar.get']('idstools:config:oinkcode', '' ) -%}
+{%- set ISAIRGAP = salt['pillar.get']('global:airgap', 'False') -%}
+{%- set MANAGERIP = salt['pillar.get']('global:managerip', '') -%}
+{%- if ISAIRGAP is sameas true -%}
+--merged=/opt/so/rules/nids/all.rules
+--local=/opt/so/rules/nids/local.rules
+--url=http://{{ MANAGERIP }}:7788/rules/emerging-all.rules
+--disable=/opt/so/idstools/etc/disable.conf
+--enable=/opt/so/idstools/etc/enable.conf
+--modify=/opt/so/idstools/etc/modify.conf
+{%- else -%}
 --suricata-version=5.0
 --merged=/opt/so/rules/nids/all.rules
 --local=/opt/so/rules/nids/local.rules
 --disable=/opt/so/idstools/etc/disable.conf
 --enable=/opt/so/idstools/etc/enable.conf
 --modify=/opt/so/idstools/etc/modify.conf
-{%- if RULESET == 'ETOPEN' %}
+  {%- if RULESET == 'ETOPEN' -%}
 --etopen
-{%- elif RULESET == 'ETPRO' %}
+  {%- elif RULESET == 'ETPRO' -%}
 --etpro={{ OINKCODE }}
-{%- elif RULESET == 'TALOS' %}
+  {%- elif RULESET == 'TALOS' -%}
 --url=https://www.snort.org/rules/snortrules-snapshot-2983.tar.gz?oinkcode={{ OINKCODE }}
-{%- endif %}
-{%- if URLS != None %}
-{%- for URL in URLS %}
+  {%- endif -%}
+{%- endif -%}
+{%- if URLS != None -%}
+{%- for URL in URLS -%}
 --url={{ URL }}
-{%- endfor %}
-{%- endif %}
+{%- endfor -%}
+{%- endif -%}

salt/idstools/init.sls

@@ -12,8 +12,13 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'idstools' in top_states %}
+
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 {% set MANAGER = salt['grains.get']('master') %}
 # IDSTools Setup
 idstoolsdir:
@@ -55,7 +60,7 @@ rulesdir:
 synclocalnidsrules:
   file.managed:
     - name: /opt/so/rules/nids/local.rules
-    - source: salt://idstools/localrules/local.rules
+    - source: salt://idstools/local.rules
     - user: 939
     - group: 939
 
@@ -69,3 +74,11 @@ so-idstools:
       - /opt/so/rules/nids:/opt/so/rules/nids:rw
     - watch:
       - file: idstoolsetcsync
+
+{% else %}
+
+idstools_state_not_allowed:
+  test.fail_without_changes:
+    - name: idstools_state_not_allowed
+
+{% endif%}

salt/idstools/localrules/local.rules

@@ -1 +0,0 @@
-# Put your own custom Snort/Suricata rules in /opt/so/saltstack/local/salt/idstools/localrules/.

salt/influxdb/init.sls

@@ -1,7 +1,12 @@
+{% set show_top = salt['state.show_top']() %}
+{% set top_states = show_top.values() | join(', ') %}
+
+{% if 'influxdb' in top_states %}
+
 {% set GRAFANA = salt['pillar.get']('manager:grafana', '0') %}
 {% set MANAGER = salt['grains.get']('master') %}
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.2') %}
-{% set IMAGEREPO = salt['pillar.get']('static:imagerepo') %}
+{% set VERSION = salt['pillar.get']('global:soversion', 'HH1.2.2') %}
+{% set IMAGEREPO = salt['pillar.get']('global:imagerepo') %}
 
 {% if grains['role'] in ['so-manager', 'so-managersearch', 'so-eval', 'so-standalone'] and GRAFANA == 1 %}
 
@@ -40,4 +45,12 @@ so-influxdb:
     - watch:
       - file: influxdbconf
 
+{% endif %}
+
+{% else %}
+
+influxdb_state_not_allowed:
+  test.fail_without_changes:
+    - name: influxdb_state_not_allowed
+
 {% endif %}

salt/kibana/bin/so-kibana-config-load

@@ -1,9 +1,7 @@
 #!/bin/bash
-# {%- set FLEET_MANAGER = salt['pillar.get']('static:fleet_manager', False) -%}
-# {%- set FLEET_NODE = salt['pillar.get']('static:fleet_node', False) -%}
-# {%- set MANAGER = salt['pillar.get']('manager:url_base', '') %}
-
-KIBANA_VERSION="7.6.1"
+# {%- set FLEET_MANAGER = salt['pillar.get']('global:fleet_manager', False) -%}
+# {%- set FLEET_NODE = salt['pillar.get']('global:fleet_node', False) -%}
+# {%- set MANAGER = salt['pillar.get']('global:url_base', '') %}
 
 # Copy template file
 cp /opt/so/conf/kibana/saved_objects.ndjson.template /opt/so/conf/kibana/saved_objects.ndjson
@@ -17,4 +15,4 @@ cp /opt/so/conf/kibana/saved_objects.ndjson.template /opt/so/conf/kibana/saved_o
 sed -i "s/PLACEHOLDER/{{ MANAGER }}/g" /opt/so/conf/kibana/saved_objects.ndjson
 
 # Load saved objects
-curl -X POST "localhost:5601/api/saved_objects/_import" -H "kbn-xsrf: true" --form file=@/opt/so/conf/kibana/saved_objects.ndjson > /dev/null 2>&1
+curl -X POST "localhost:5601/api/saved_objects/_import?overwrite=true" -H "kbn-xsrf: true" --form file=@/opt/so/conf/kibana/saved_objects.ndjson > /dev/null 2>&1

salt/kibana/etc/kibana.yml

@@ -1,6 +1,7 @@
 ---
 # Default Kibana configuration from kibana-docker.
 {%- set ES = salt['pillar.get']('manager:mainip', '') -%}
+{%- set FEATURES = salt['pillar.get']('elastic:features', False) %}
 server.name: kibana
 server.host: "0"
 server.basePath: /kibana