Merge pull request #586 from Security-Onion-Solutions/dev

1.2.1-1

README.md

@@ -1,37 +1,39 @@
-## Hybrid Hunter Alpha 1.1.4 - Feature Parity Release
+## Hybrid Hunter Beta 1.2.1 - Beta 1
 
 ### Changes:
 
-- Added new in-house auth method [Security Onion Auth](https://github.com/Security-Onion-Solutions/securityonion-auth).
-- Web user creation is done via the browser now instead of so-user-add.
-- New Logstash pipeline setup. Now uses multiple pipelines.
-- New Master + Search node type and well as a Heavy Node type in the install. 
-- Change all nodes to point to the docker registry on the Master. This cuts down on the calls to dockerhub.
-- Zeek 3.0.1
-- Elastic 6.8.6
-- New SO Start | Stop | Restart scripts for all components (eg. `so-playbook-restart`).
-- BPF support for Suricata (NIDS), Steno (PCAP) & Zeek ([Docs](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/BPF)).
-- Updated Domain Stats & Frequency Server containers to Python3 & created new Salt states for them.
-- Added so-status script which gives an easy to read look at container status.
-- Manage threshold.conf for Suricata using the thresholding pillar.
-- The ISO now includes all the docker containers for faster install speeds.
-- You now set the password for the onion account during the iso install. This account is temporary and will be removed after so-setup. 
-- Updated Helix parsers for better compatibility.
-- Updated telegraf docker to include curl and jq.
-- CVE-2020-0601 Zeek Detection Script. 
-- ISO Install now prompts you to create a password for the onion user during imaging. This account gets disabled during setup.
+- Full support for Ubuntu 18.04. 16.04 is no longer supported for Hybrid Hunter.
+- Introduction of the Security Onion Console. Once logged in you are directly taken to the SOC.
+- New authentication using Kratos.
+- During install you must specify how you would like to access the SOC ui. This is for strict cookie security.
+- Ability to list and delete web users from the SOC ui.
+- The soremote account is now used to add nodes to the grid vs using socore. 
+- Community ID support for Zeek, osquery, and Suricata. You can now tie host events to connection logs!
+- Elastic 7.6.1 with ECS support.
+- New set of Kibana dashboards that align with ECS.
+- Eval mode no longer uses Logstash for parsing (Filebeat -> ES Ingest)
+- Ingest node parsing for osquery-shipped logs (osquery, WEL, Sysmon).
+- Fleet standalone mode with improved Web UI & API access control.
+- Improved Fleet integration support.
+- Playbook now has full Windows Sigma community ruleset builtin.
+- Automatic Sigma community rule updates.
+- Playbook stability enhancements.
+- Zeek health check. Zeek will now auto restart if a worker crashes.
+- zeekctl is now managed by salt.
+- Grafana dashboard improvements and cleanup.
+- Moved logstash configs to pillars.
+- Salt logs moved to /opt/so/log/salt.
+- Strelka integrated for file-oriented detection/analysis at scale
 
-## Version 1.1.4 ISO Download
+### Known issues:
 
-[HH1.1.4-44.ISO](https://download.securityonion.net/file/Hybrid-Hunter/HH-1.1.4-44.iso)  
-
-MD5: C881536D55C5791F69596E474513E953  
-SHA1: 1CF503A46279EDDC5C84AA9F02167839004E7723  
-SHA256: F5C2FB52DFD314540019953BFE1960AF130AB09CD75E60E66CAA122DAD4DA414  
+- Updating users via the SOC ui is known to fail. To change a user, delete the user and re-add them. 
+- Due to the move to ECS, the current Playbook plays may not alert correctly at this time.
+- The osquery MacOS package does not install correctly.
 
 ### Warnings and Disclaimers
 
-- This ALPHA release is BLEEDING EDGE and TOTALLY UNSUPPORTED!  
+- This BETA release is BLEEDING EDGE and TOTALLY UNSUPPORTED!  
 - If this breaks your system, you get to keep both pieces!  
 - This script is a work in progress and is in constant flux.  
 - This script is intended to build a quick prototype proof of concept so you can see what our new platform might look like.  This configuration will change drastically over time leading up to the final release.  
@@ -44,33 +46,36 @@ SHA256: F5C2FB52DFD314540019953BFE1960AF130AB09CD75E60E66CAA122DAD4DA414
 
 Evaluation Mode:
 
-- ISO or a Single VM running Ubuntu 16.04 or CentOS 7
+- ISO or a Single VM running Ubuntu 18.04 or CentOS 7
 - Minimum 12GB of RAM
 - Minimum 4 CPU cores
 - Minimum 2 NICs
 
 Distributed:
 
-- 3 VMs running the ISO or Ubuntu 16.04 or CentOS 7 (You can mix and match)
+- 3 VMs running the ISO or Ubuntu 18.04 or CentOS 7 (You can mix and match)
 - Minimum 8GB of RAM per VM
 - Minimum 4 CPU cores per VM
 - Minimum 2 NICs for forward nodes
 
-### Prerequisites for Network Based Install
+### Installation
 
-Install git if using a Centos 7 Minimal install:
+For most users, we recommend installing using [our ISO image](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/ISO).
+
+If instead you would like to try a manual installation (not using our ISO), you can build from CentOS 7 or Ubuntu 18.04.
+
+If using CentOS 7 Minimal, you will need to install git:
 
 ```sudo yum -y install git```
 
-### Installation
-
-Once you resolve those requirements or are using Ubuntu 16.04 do the following:
+Once you have git, then do the following:
 
 ```
 git clone https://github.com/Security-Onion-Solutions/securityonion-saltstack
 cd securityonion-saltstack
-sudo bash so-setup-network.sh
+sudo bash so-setup-network
 ```
+
 Follow the prompts and reboot if asked to do so.
 
 Then proceed to the [Hybrid Hunter Quick Start Guide](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/Hybrid-Hunter-Quick-Start-Guide).

VERSION

@@ -1 +1 @@
-1.0.6
+1.2.1

exclude-list.txt

@@ -1,2 +0,0 @@
-salt/bro/files/local.bro
-salt/bro/files/local.bro.community

files/master

@@ -12,6 +12,7 @@
 # modified files cause conflicts, set verify_env to False.
 # user: socore
 
+log_file: /opt/so/log/salt/master
 
 #####      File Server settings      #####
 ##########################################
@@ -57,3 +58,7 @@ pillar_roots:
 peer:
   .*:
     - x509.sign_remote_certificate
+
+reactor:
+  - 'so/fleet':
+    - salt://reactor/fleet.sls

pillar/docker/config.sls

@@ -1,4 +1,5 @@
-{% set OSQUERY = salt['pillar.get']('master:osquery', '0') %}
+{%- set FLEETMASTER = salt['pillar.get']('static:fleet_master', False) -%}
+{%- set FLEETNODE = salt['pillar.get']('static:fleet_node', False) -%}
 {% set WAZUH = salt['pillar.get']('master:wazuh', '0') %}
 {% set THEHIVE = salt['pillar.get']('master:thehive', '0') %}
 {% set PLAYBOOK = salt['pillar.get']('master:playbook', '0') %}
@@ -7,7 +8,6 @@
 {% set BROVER = salt['pillar.get']('static:broversion', 'COMMUNITY') %}
 {% set GRAFANA = salt['pillar.get']('master:grafana', '0') %}
 
-
 eval:
   containers:
     - so-core
@@ -17,11 +17,10 @@ eval:
     - so-grafana
     {% endif %}
     - so-dockerregistry
-    - so-sensoroni
+    - so-soc
+    - so-kratos
     - so-idstools
-    - so-auth-api
-    - so-auth-ui
-    {% if OSQUERY != '0' %}
+    {% if FLEETMASTER %}
     - so-mysql
     - so-fleet
     - so-redis
@@ -89,12 +88,11 @@ master_search:
   containers:
     - so-core
     - so-telegraf
-    - so-sensoroni
+    - so-soc
+    - so-kratos
     - so-acng
     - so-idstools
     - so-redis
-    - so-auth-api
-    - so-auth-ui
     - so-logstash
     - so-elasticsearch
     - so-curator
@@ -102,7 +100,7 @@ master_search:
     - so-elastalert
     - so-filebeat
     - so-soctopus
-    {% if OSQUERY != '0' %}
+    {% if FLEETMASTER %}
     - so-mysql
     - so-fleet
     - so-redis
@@ -135,18 +133,17 @@ master:
     - so-influxdb
     - so-grafana
     {% endif %}
-    - so-sensoroni
+    - so-soc
+    - so-kratos
     - so-acng
     - so-idstools
     - so-redis
-    - so-auth-api
-    - so-auth-ui
     - so-elasticsearch
     - so-logstash
     - so-kibana
     - so-elastalert
     - so-filebeat
-    {% if OSQUERY != '0' %}
+    {% if FLEETMASTER %}
     - so-mysql
     - so-fleet
     - so-redis
@@ -202,4 +199,13 @@ warm_node:
     - so-core
     - so-telegraf
     - so-elasticsearch
-    
+fleet:
+  containers:
+    {% if FLEETNODE %}
+    - so-mysql
+    - so-fleet
+    - so-redis
+    - so-filebeat
+    - so-core
+    - so-telegraf
+    {% endif %}

pillar/healthcheck/eval.sls

@@ -0,0 +1,5 @@
+healthcheck:
+  enabled: False
+  schedule: 300
+  checks:
+    - zeek

pillar/healthcheck/sensor.sls

@@ -0,0 +1,5 @@
+healthcheck:
+  enabled: False
+  schedule: 300
+  checks:
+    - zeek

pillar/logstash/eval.sls

@@ -1,4 +1,21 @@
 logstash:
   pipelines:
     eval:
-      config: "/usr/share/logstash/pipelines/eval/*.conf"
+      config:
+        - so/0800_input_eval.conf
+        - so/1002_preprocess_json.conf
+        - so/1033_preprocess_snort.conf
+        - so/7100_osquery_wel.conf
+        - so/8999_postprocess_rename_type.conf
+        - so/9000_output_bro.conf.jinja
+        - so/9002_output_import.conf.jinja
+        - so/9033_output_snort.conf.jinja
+        - so/9100_output_osquery.conf.jinja
+        - so/9400_output_suricata.conf.jinja
+        - so/9500_output_beats.conf.jinja
+        - so/9600_output_ossec.conf.jinja
+        - so/9700_output_strelka.conf.jinja
+  templates:
+    - so/so-beats-template.json
+    - so/so-common-template.json
+    - so/so-zeek-template.json

pillar/logstash/helix.sls

@@ -1,4 +1,42 @@
 logstash:
   pipelines:
     helix:
-      config: "/usr/share/logstash/pipelines/helix/*.conf"
+      config:
+        - so/0010_input_hhbeats.conf
+        - so/1033_preprocess_snort.conf
+        - so/1100_preprocess_bro_conn.conf
+        - so/1101_preprocess_bro_dhcp.conf
+        - so/1102_preprocess_bro_dns.conf
+        - so/1103_preprocess_bro_dpd.conf
+        - so/1104_preprocess_bro_files.conf
+        - so/1105_preprocess_bro_ftp.conf
+        - so/1106_preprocess_bro_http.conf
+        - so/1107_preprocess_bro_irc.conf
+        - so/1108_preprocess_bro_kerberos.conf
+        - so/1109_preprocess_bro_notice.conf
+        - so/1110_preprocess_bro_rdp.conf
+        - so/1111_preprocess_bro_signatures.conf
+        - so/1112_preprocess_bro_smtp.conf
+        - so/1113_preprocess_bro_snmp.conf
+        - so/1114_preprocess_bro_software.conf
+        - so/1115_preprocess_bro_ssh.conf
+        - so/1116_preprocess_bro_ssl.conf
+        - so/1117_preprocess_bro_syslog.conf
+        - so/1118_preprocess_bro_tunnel.conf
+        - so/1119_preprocess_bro_weird.conf
+        - so/1121_preprocess_bro_mysql.conf
+        - so/1122_preprocess_bro_socks.conf
+        - so/1123_preprocess_bro_x509.conf
+        - so/1124_preprocess_bro_intel.conf
+        - so/1125_preprocess_bro_modbus.conf
+        - so/1126_preprocess_bro_sip.conf
+        - so/1127_preprocess_bro_radius.conf
+        - so/1128_preprocess_bro_pe.conf
+        - so/1129_preprocess_bro_rfb.conf
+        - so/1130_preprocess_bro_dnp3.conf
+        - so/1131_preprocess_bro_smb_files.conf
+        - so/1132_preprocess_bro_smb_mapping.conf
+        - so/1133_preprocess_bro_ntlm.conf
+        - so/1134_preprocess_bro_dce_rpc.conf
+        - so/8001_postprocess_common_ip_augmentation.conf
+        - so/9997_output_helix.conf.jinja

pillar/logstash/init.sls

@@ -0,0 +1,11 @@
+logstash:
+  docker_options:
+    port_bindings:
+      - 0.0.0.0:514:514
+      - 0.0.0.0:5044:5044
+      - 0.0.0.0:5644:5644
+      - 0.0.0.0:6050:6050
+      - 0.0.0.0:6051:6051
+      - 0.0.0.0:6052:6052
+      - 0.0.0.0:6053:6053
+      - 0.0.0.0:9600:9600

pillar/logstash/master.sls

@@ -1,4 +1,6 @@
 logstash:
   pipelines:
     master:
-      config: "/usr/share/logstash/pipelines/master/*.conf"
+      config:
+        - so/0010_input_hhbeats.conf
+        - so/9999_output_redis.conf.jinja

pillar/logstash/search.sls

@@ -1,4 +1,16 @@
 logstash:
   pipelines:
     search:
-      config: "/usr/share/logstash/pipelines/search/*.conf"
+      config:
+        - so/0900_input_redis.conf.jinja
+        - so/9000_output_zeek.conf.jinja
+        - so/9002_output_import.conf.jinja
+        - so/9100_output_osquery.conf.jinja
+        - so/9400_output_suricata.conf.jinja
+        - so/9500_output_beats.conf.jinja
+        - so/9600_output_ossec.conf.jinja
+        - so/9700_output_strelka.conf.jinja
+  templates:
+    - so/so-beats-template.json
+    - so/so-common-template.json
+    - so/so-zeek-template.json

pillar/masters/example.sls

@@ -1,10 +0,0 @@
-# Example Pillar file for a master
-master:
-  esaccessip: 127.0.0.1
-  esheap: CHANGEME
-  esclustername: {{ grains.host }}
-  freq: 0
-  domainstats: 0
-  lsheap: 1500m
-  lsaccessip: 127.0.0.1
-  elastalert: 1

pillar/nodes/example.sls

@@ -1,5 +0,0 @@
-# Example Pillar file for a sensor
-node:
-  ls_heapsize: CHANGEME
-  es_heapsize: CHANGEME
-  node_type: CHANGEME

pillar/sensors/example.sls

@@ -1,14 +0,0 @@
-# Example Pillar file for a sensor
-sensor:
-  interface: CHANGEME
-  bro_pins:
-    - 1
-    - 2
-    - 3
-    - 4
-  brobpf:
-  pcapbpf:
-  nidsbpf:
-  s3bucket:
-  s3key:
-

pillar/top.sls

@@ -3,53 +3,70 @@ base:
     - patch.needs_restarting
     - docker.config
 
-  'G@role:so-mastersearch or G@role:so-heavynode':
+  '*_mastersearch or *_heavynode':
     - match: compound
+    - logstash
     - logstash.master
     - logstash.search
 
-  'G@role:so-sensor':
+  '*_sensor':
     - static
     - firewall.*
     - brologs
+    - healthcheck.sensor
     - minions.{{ grains.id }}
 
-  'G@role:so-master or G@role:so-mastersearch':
+  '*_master or *_mastersearch':
     - match: compound
     - static
     - firewall.*
     - data.*
-    - auth
+    - secrets
     - minions.{{ grains.id }}
 
-  'G@role:so-master':
+  '*_master':
+    - logstash
     - logstash.master
 
-  'G@role:so-eval':
+  '*_eval':
     - static
     - firewall.*
     - data.*
     - brologs
-    - auth
-    - logstash.eval
+    - secrets
+    - healthcheck.eval
     - minions.{{ grains.id }}
 
-  'G@role:so-node':
+  '*_node':
     - static
     - firewall.*
     - minions.{{ grains.id }}
 
-  'G@role:so-heavynode':
+  '*_heavynode':
     - static
     - firewall.*
     - brologs
     - minions.{{ grains.id }}
 
-  'G@role:so-helix':
+  '*_helix':
     - static
     - firewall.*
     - fireeye
     - brologs
+    - logstash
     - logstash.helix
-    - static
+    - minions.{{ grains.id }}
+
+  '*_fleet':
+    - static
+    - firewall.*
+    - data.*
+    - secrets
+    - minions.{{ grains.id }}
+
+  '*_searchnode':
+    - static
+    - firewall.*
+    - logstash
+    - logstash.search
     - minions.{{ grains.id }}

salt/_beacons/zeek.py

@@ -0,0 +1,33 @@
+import logging
+
+
+def status():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl status'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ logging.info('zeekctl_module: zeekctl.status retval: %s' % retval)
+
+ return retval
+
+
+def beacon(config):
+
+  retval = []
+
+  is_enabled = __salt__['healthcheck.is_enabled']()
+  logging.info('zeek_beacon: healthcheck_is_enabled: %s' % is_enabled)
+
+  if is_enabled:
+    zeekstatus = status().lower().split(' ')
+    logging.info('zeek_beacon: zeekctl.status: %s' % str(zeekstatus))
+    if 'stopped' in zeekstatus or 'crashed' in zeekstatus or 'error' in zeekstatus or 'error:' in zeekstatus:
+     zeek_restart = True
+    else:
+     zeek_restart = False
+
+    __salt__['telegraf.send']('healthcheck zeek_restart=%s' % str(zeek_restart))
+    retval.append({'zeek_restart': zeek_restart})
+    logging.info('zeek_beacon: retval: %s' % str(retval))
+
+  return retval
+

salt/_modules/healthcheck.py

@@ -0,0 +1,96 @@
+#!py
+
+import logging
+import sys
+
+allowed_functions = ['is_enabled', 'zeek']
+states_to_apply = []
+
+
+def apply_states(states=''):
+
+  calling_func = sys._getframe().f_back.f_code.co_name
+  logging.debug('healthcheck_module: apply_states function caller: %s' % calling_func)
+
+  if not states:
+    states = ','.join(states_to_apply)
+ 
+  if states: 
+    logging.info('healthcheck_module: apply_states states: %s' % str(states))
+    __salt__['state.apply'](states)
+
+
+def docker_stop(container):
+  
+  try:
+    stopdocker = __salt__['docker.rm'](container, 'stop=True')
+  except Exception as e:
+    logging.error('healthcheck_module: %s' % e)
+
+
+def is_enabled():
+  
+  if __salt__['pillar.get']('healthcheck:enabled', 'False'):
+    retval = True
+  else:
+    retval = False
+  
+  return retval
+  
+
+def run(checks=''):
+  
+  retval = []
+  calling_func = sys._getframe().f_back.f_code.co_name
+  logging.debug('healthcheck_module: run function caller: %s' % calling_func)
+  
+  if checks:
+    checks = checks.split(',')
+  else:  
+    checks = __salt__['pillar.get']('healthcheck:checks', {})
+  
+  logging.debug('healthcheck_module: run checks to be run: %s' % str(checks))
+  for check in checks:
+    if check in allowed_functions:
+      retval.append(check)
+      check = getattr(sys.modules[__name__], check)
+      check()
+    else:
+      logging.warning('healthcheck_module: attempted to run function %s' % check)
+
+  # If you want to apply states at the end of the run,
+  # be sure to append the state name to states_to_apply[]
+  apply_states()
+
+  return retval
+
+
+def send_event(tag, eventdata):
+  __salt__['event.send'](tag, eventdata[0])
+
+
+def zeek():
+
+  calling_func = sys._getframe().f_back.f_code.co_name
+  logging.debug('healthcheck_module: zeek function caller: %s' % calling_func)
+  retval = []
+
+  retcode = __salt__['zeekctl.status'](verbose=False)
+  logging.debug('healthcheck_module: zeekctl.status retcode: %i' % retcode)
+  if retcode:
+    zeek_restart = 1
+    if calling_func != 'beacon':
+      docker_stop('so-zeek')
+      states_to_apply.append('zeek')
+  else:
+    zeek_restart = 0
+
+  __salt__['telegraf.send']('healthcheck zeek_restart=%i' % zeek_restart)
+  
+  if calling_func == 'execute' and zeek_restart:
+    apply_states()
+  
+  retval.append({'zeek_restart': zeek_restart})
+
+  send_event('so/healthcheck/zeek', retval)
+  return retval

salt/_modules/telegraf.py

@@ -0,0 +1,16 @@
+#!py
+
+import logging
+import socket
+
+
+def send(data):
+
+  mainint = __salt__['pillar.get']('sensor:mainint', __salt__['pillar.get']('master:mainint'))
+  mainip = __salt__['grains.get']('ip_interfaces').get(mainint)[0]
+  dstport = 8094
+
+  sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+  sent = sock.sendto(data.encode('utf-8'), (mainip, dstport))
+
+  return sent

salt/_modules/zeekctl.py

@@ -0,0 +1,160 @@
+#!py
+
+import logging
+
+
+def capstats(interval=10):
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl capstats %i'" % interval
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ 
+ return retval
+
+
+def check():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl check'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ 
+ return retval
+
+
+def cleanup(all=''):
+
+ retval = ''
+
+ if all: 
+   if all == 'all':
+     cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl cleanup --all'"
+   else:
+     retval = 'Invalid option. zeekctl.help for options'
+ else:
+   cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl cleanup'"
+
+ if not retval:
+   retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def config():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl config'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def deploy():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl deploy'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def df():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl df'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def diag():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl diag'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def install(local=''):
+
+ retval = ''
+
+ if local:
+   if local == 'local':
+     cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl install --local'"
+   else:
+     retval = 'Invalid option. zeekctl.help for options' 
+ else:
+   cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl install'"
+ 
+ if not retval:
+   retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def netstats():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl netstats'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def nodes():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl nodes'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def restart(clean=''):
+
+ retval = ''
+
+ if clean:
+   if clean == 'clean':
+     cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl restart --clean'"
+   else:
+     retval = 'Invalid option. zeekctl.help for options'
+ else:
+   cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl restart'"
+ 
+ if not retval:
+   retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def scripts(c=''):
+
+ retval = ''
+
+ if c:
+   if c == 'c':
+     cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl scripts -c'"
+   else:
+     retval = 'Invalid option. zeekctl.help for options'
+ else:
+   cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl scripts'"
+
+ if not retval:
+   retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def start():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl start'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def status(verbose=True):
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl status'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ if not verbose:
+   retval = __context__['retcode']
+ logging.info('zeekctl_module: zeekctl.status retval: %s' % retval)
+ return retval
+
+
+def stop():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl stop'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval
+
+
+def top():
+
+ cmd = "runuser -l zeek -c '/opt/zeek/bin/zeekctl top'"
+ retval = __salt__['docker.run']('so-zeek', cmd)
+ return retval

salt/auth/init.sls

@@ -1,30 +0,0 @@
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.1.4') %}
-{% set MASTER = salt['grains.get']('master') %}
-
-so-auth-api-dir:
-  file.directory:
-    - name: /opt/so/conf/auth/api
-    - user: 939
-    - group: 939
-    - makedirs: True
-
-so-auth-api:
-    docker_container.running:
-        - image: {{ MASTER }}:5000/soshybridhunter/so-auth-api:{{ VERSION }}
-        - hostname: so-auth-api
-        - name: so-auth-api
-        - environment:
-            - BASE_PATH: "/so-auth/api"
-            - AUTH_TOKEN_TIMEOUT: 32400
-        - binds:
-            - /opt/so/conf/auth/api:/data
-        - port_bindings:
-            - 0.0.0.0:5656:5656
-
-so-auth-ui:
-    docker_container.running:
-        - image: {{ MASTER }}:5000/soshybridhunter/so-auth-ui:{{ VERSION }}
-        - hostname: so-auth-ui
-        - name: so-auth-ui
-        - port_bindings:
-            - 0.0.0.0:4242:80

salt/common/grafana/etc/dashboards/dashboard.yml

@@ -10,13 +10,13 @@ providers:
   editable: true
   options:
     path: /etc/grafana/grafana_dashboards/master
-- name: 'Forward Nodes'
-  folder: 'Forward Nodes'
+- name: 'Sensor Nodes'
+  folder: 'Sensor Nodes'
   type: file
   disableDeletion: false
   editable: true
   options:
-    path: /etc/grafana/grafana_dashboards/forward_nodes
+    path: /etc/grafana/grafana_dashboards/sensor_nodes
 - name: 'Search Nodes'
   folder: 'Search Nodes'
   type: file

salt/common/init.sls

@@ -1,6 +1,8 @@
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.1.4') %}
+{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.1') %}
 {% set MASTER = salt['grains.get']('master') %}
 {% set GRAFANA = salt['pillar.get']('master:grafana', '0') %}
+{% set FLEETMASTER = salt['pillar.get']('static:fleet_master', False) %}
+{% set FLEETNODE = salt['pillar.get']('static:fleet_node', False) %}
 # Add socore Group
 socoregroup:
   group.present:
@@ -81,10 +83,6 @@ docker:
   service.running:
     - enable: True
 
-salt-minion:
-  service.running:
-    - enable: True
-
 # Drop the correct nginx config based on role
 
 nginxconfdir:
@@ -102,13 +100,6 @@ nginxconf:
     - template: jinja
     - source: salt://common/nginx/nginx.conf.{{ grains.role }}
 
-copyindex:
-  file.managed:
-    - name: /opt/so/conf/nginx/index.html
-    - user: 939
-    - group: 939
-    - source: salt://common/nginx/index.html
-
 nginxlogdir:
   file.directory:
     - name: /opt/so/log/nginx/
@@ -131,7 +122,6 @@ so-core:
     - binds:
       - /opt/so:/opt/so:rw
       - /opt/so/conf/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
-      - /opt/so/conf/nginx/index.html:/opt/socore/html/index.html:ro
       - /opt/so/log/nginx/:/var/log/nginx:rw
       - /opt/so/tmp/nginx/:/var/lib/nginx:rw
       - /opt/so/tmp/nginx/:/run:rw
@@ -142,6 +132,9 @@ so-core:
     - port_bindings:
       - 80:80
       - 443:443
+    {%- if FLEETMASTER or FLEETNODE %}
+      - 8090:8090
+    {%- endif %}
     - watch:
       - file: /opt/so/conf/nginx/nginx.conf
 
@@ -187,6 +180,8 @@ so-telegraf:
       - HOST_SYS=/host/sys
       - HOST_MOUNT_PREFIX=/host
     - network_mode: host
+    - port_bindings:
+      - 127.0.0.1:8094:8094
     - binds:
       - /opt/so/log/telegraf:/var/log/telegraf:rw
       - /opt/so/conf/telegraf/etc/telegraf.conf:/etc/telegraf/telegraf.conf:ro
@@ -212,7 +207,7 @@ so-telegraf:
       - /opt/so/conf/telegraf/scripts
 
 # If its a master or eval lets install the back end for now
-{% if grains['role'] == 'so-master' or grains['role'] == 'so-eval' and GRAFANA == 1 %}
+{% if grains['role'] in ['so-master', 'so-mastersearch', 'so-eval'] and GRAFANA == 1 %}
 
 # Influx DB
 influxconfdir:
@@ -287,7 +282,7 @@ grafanadashevaldir:
 
 grafanadashfndir:
   file.directory:
-    - name: /opt/so/conf/grafana/grafana_dashboards/forward_nodes
+    - name: /opt/so/conf/grafana/grafana_dashboards/sensor_nodes
     - user: 939
     - group: 939
     - makedirs: True
@@ -308,7 +303,9 @@ grafanaconf:
     - source: salt://common/grafana/etc
 
 {% if salt['pillar.get']('mastertab', False) %}
-{%- for SN, SNDATA in salt['pillar.get']('mastertab', {}).items() %}
+{% for SN, SNDATA in salt['pillar.get']('mastertab', {}).items() %}
+{% set NODETYPE = SN.split('_')|last %}
+{% set SN = SN | regex_replace('_' ~ NODETYPE, '') %}
 dashboard-master:
   file.managed:
     - name: /opt/so/conf/grafana/grafana_dashboards/master/{{ SN }}-Master.json
@@ -325,18 +322,20 @@ dashboard-master:
       ROOTFS: {{ SNDATA.rootfs }}
       NSMFS: {{ SNDATA.nsmfs }}
 
-{%- endfor %}
+{% endfor %}
 {% endif %}
 
 {% if salt['pillar.get']('sensorstab', False) %}
-{%- for SN, SNDATA in salt['pillar.get']('sensorstab', {}).items() %}
+{% for SN, SNDATA in salt['pillar.get']('sensorstab', {}).items() %}
+{% set NODETYPE = SN.split('_')|last %}
+{% set SN = SN | regex_replace('_' ~ NODETYPE, '') %}
 dashboard-{{ SN }}:
   file.managed:
-    - name: /opt/so/conf/grafana/grafana_dashboards/forward_nodes/{{ SN }}-Sensor.json
+    - name: /opt/so/conf/grafana/grafana_dashboards/sensor_nodes/{{ SN }}-Sensor.json
     - user: 939
     - group: 939
     - template: jinja
-    - source: salt://common/grafana/grafana_dashboards/forward_nodes/sensor.json
+    - source: salt://common/grafana/grafana_dashboards/sensor_nodes/sensor.json
     - defaults:
       SERVERNAME: {{ SN }}
       MONINT: {{ SNDATA.monint }}
@@ -350,7 +349,9 @@ dashboard-{{ SN }}:
 {% endif %}
 
 {% if salt['pillar.get']('nodestab', False) %}
-{%- for SN, SNDATA in salt['pillar.get']('nodestab', {}).items() %}
+{% for SN, SNDATA in salt['pillar.get']('nodestab', {}).items() %}
+{% set NODETYPE = SN.split('_')|last %}
+{% set SN = SN | regex_replace('_' ~ NODETYPE, '') %}
 dashboardsearch-{{ SN }}:
   file.managed:
     - name: /opt/so/conf/grafana/grafana_dashboards/search_nodes/{{ SN }}-Node.json
@@ -371,7 +372,9 @@ dashboardsearch-{{ SN }}:
 {% endif %}
 
 {% if salt['pillar.get']('evaltab', False) %}
-{%- for SN, SNDATA in salt['pillar.get']('evaltab', {}).items() %}
+{% for SN, SNDATA in salt['pillar.get']('evaltab', {}).items() %}
+{% set NODETYPE = SN.split('_')|last %}
+{% set SN = SN | regex_replace('_' ~ NODETYPE, '') %}
 dashboard-{{ SN }}:
   file.managed:
     - name: /opt/so/conf/grafana/grafana_dashboards/eval/{{ SN }}-Node.json

salt/common/nginx/index.html

@@ -1,130 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-<title>Security Onion - Hybrid Hunter</title>
-<meta charset="utf-8">
-<meta name="viewport" content="width=device-width, initial-scale=1">
-<link rel="icon" type="image/png" href="favicon-32x32.png" sizes="32x32" />
-<link rel="icon" type="image/png" href="favicon-16x16.png" sizes="16x16" />
-<style>
-* {
-  box-sizing: border-box;
-  font-family: Arial, Helvetica, sans-serif;
-	padding-left: 30px;
-	padding-right: 30px;
-}
-
-body {
-  font-family: Arial, Helvetica, sans-serif;
-	background-color: #2a2a2a;
-
-}
-a {
-	color: #f2f2f2;
-	text-align: left;
-	padding: 0px;
-}
-
-.center {
-	margin: 0 auto;
-}
-
-/* Style the top navigation bar */
-.topnav {
-  overflow: hidden;
-  background-color: #333;
-  width: 1080px;
-  display: flex;
-  align-content: center;
-}
-
-/* Style the topnav links */
-.topnav a {
-  margin: auto;
-  color: #f2f2f2;
-  text-align: center;
-  padding: 14px 16px;
-  text-decoration: none;
-}
-
-/* Change color on hover */
-.topnav a:hover {
-  background-color: #ddd;
-  color: black;
-}
-
-/* Style the content */
-.content {
-  background-color: #2a2a2a;
-  padding: 10px;
-	padding-top: 20px;
-	padding-left: 60px;
-	color: #E3DBCC;
-	width: 1080px;
-}
-
-/* Style the footer */
-.footer {
-  background-color: #2a2a2a;
-  padding: 60px;
-  color: #E3DBCC;
-  width: 1080px;
-}
-
-</style>
-</head>
-<body>
-	<div class="center">
-		<div class="topnav center">
-			<a href="/so-auth/loginpage/create-user" target="_blank">Create New User</a>
-			<a href="/kibana/" target="_blank">Kibana</a>
-			<a href="/grafana/" target="_blank">Grafana</a>
-			<a href="/sensoroni/" target="_blank">Sensoroni</a>
-	  		<a href="/playbook/" target="_blank">Playbook</a>
-			<a href="/fleet/" target="_blank">Fleet</a>
-			<a href="/thehive/" target="_blank">TheHive</a>
-			<a href="/packages/" target="_blank">Osquery Packages</a>
-			<a href="https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/FAQ" target="_blank">FAQ</a>
-			<a href="https://www.securityonionsolutions.com" target="_blank">Security Onion Solutions</a>
-			<a href="https://blog.securityonion.net" target="_blank">Blog</a>
-		</div>
-
-		<div class="content center">
-			<center><a href="https://securityonion.net"><img STYLE="border: none;" src="alpha_logo.jpg" alt="Security Onion" align="center" target="_blank"></img></a><br></center>
-
-			<p><center><h1>Hybrid Hunter Alpha 1.1.4 - Feature Parity Release</h1></center><br>
-				<h2>Changes:</h2>
-        <ul>
-          <li>Added new in-house auth method [Security Onion Auth](https://github.com/Security-Onion-Solutions/securityonion-auth).</li>
-          <li>Web user creation is done via the browser now instead of so-user-add.</li>
-          <li>New Logstash pipeline setup. Now uses multiple pipelines.</li>
-          <li>New Master + Search node type and well as a Heavy Node type in the install.</li>
-          <li>Change all nodes to point to the docker registry on the Master. This cuts down on the calls to dockerhub.</li>
-          <li>Zeek 3.0.1</li>
-          <li>Elastic 6.8.6</li>
-          <li>New SO Start | Stop | Restart scripts for all components (eg. `so-playbook-restart`).</li>
-          <li>BPF support for Suricata (NIDS), Steno (PCAP) & Zeek ([Docs](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/BPF)).</li>
-          <li>Updated Domain Stats & Frequency Server containers to Python3 & created new Salt states for them.</li>
-          <li>Added so-status script which gives an easy to read look at container status.</li>
-          <li>Manage threshold.conf for Suricata using the thresholding pillar.</li>
-          <li>The ISO now includes all the docker containers for faster install speeds.</li>
-          <li>You now set the password for the onion account during the iso install. This account is temporary and will be removed after so-setup.</li>
-          <li>Updated Helix parsers for better compatibility.</li>
-          <li>Updated telegraf docker to include curl and jq.</li>
-          <li>CVE-2020-0601 Zeek Detection Script.</li>
-          <li>ISO Install now prompts you to create a password for the onion user during imaging. This account gets disabled during setup.</li>
-    		  <li>Check out the <a href="https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/Hybrid-Hunter-Quick-Start-Guide" target="_blank">Hybrid Hunter Quick Start Guide</a>.</li>
-		   </ul>
-	  </p>
-		</div>
-
-		<div class="footer center">
-		<b>Disclaimer of Warranty</b><br>
-			<small>THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM .AS IS. WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.</small><br>
-			<br>
-			<b>Limitation of Liability</b><br>
-			<small>IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.</small><br>
-		</div>
-	</div>
-</body>
-</html>

salt/common/nginx/nginx.conf.so-eval

@@ -1,9 +1,11 @@
 {%- set masterip = salt['pillar.get']('master:mainip', '') %}
+{%- set FLEET_MASTER = salt['pillar.get']('static:fleet_master') %}
+{%- set FLEET_NODE = salt['pillar.get']('static:fleet_node') %}
+{%- set FLEET_IP = salt['pillar.get']('static:fleet_ip', None) %}
 # For more information on configuration, see:
 #   * Official English Documentation: http://nginx.org/en/docs/
 #   * Official Russian Documentation: http://nginx.org/ru/docs/
 
-user nginx;
 worker_processes auto;
 error_log /var/log/nginx/error.log;
 pid /run/nginx.pid;
@@ -27,6 +29,7 @@ http {
     tcp_nodelay         on;
     keepalive_timeout   65;
     types_hash_max_size 2048;
+    client_max_body_size 1024M;
 
     include             /etc/nginx/mime.types;
     default_type        application/octet-stream;
@@ -63,6 +66,29 @@ http {
         return 301 https://$host$request_uri;
     }
 
+{% if FLEET_MASTER %}
+    server {
+        listen       8090 ssl http2 default_server;
+        server_name  _;
+        root         /opt/socore/html;
+        index        blank.html;
+
+        ssl_certificate "/etc/pki/nginx/server.crt";
+        ssl_certificate_key "/etc/pki/nginx/server.key";
+        ssl_session_cache shared:SSL:1m;
+        ssl_session_timeout  10m;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+
+        location ~ ^/kolide.agent.Api/(RequestEnrollment|RequestConfig|RequestQueries|PublishLogs|PublishResults|CheckHealth)$ {
+            grpc_pass  grpcs://{{ masterip }}:8080;
+            grpc_set_header Host $host;
+            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_buffering off;
+        }
+
+    }
+{% endif %}
 
 # Settings for a TLS enabled server.
 
@@ -83,10 +109,54 @@ http {
         # Load configuration files for the default server block.
         #include /etc/nginx/default.d/*.conf;
 
-        #location / {
-    #      try_files $uri $uri.html /index.html;
-    #    }
+        location ~* (^/login/|^/js/.*|^/css/.*|^/images/.*) {
+          proxy_pass            http://{{ masterip }}:9822;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+          proxy_set_header      Upgrade $http_upgrade;
+          proxy_set_header      Connection "Upgrade";
+        }
 
+        location / {
+          auth_request          /auth/sessions/whoami;
+          proxy_pass            http://{{ masterip }}:9822/;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+          proxy_set_header      Upgrade $http_upgrade;
+          proxy_set_header      Connection "Upgrade";
+        }
+
+        location ~ ^/auth/.*?(whoami|login|logout) {
+          rewrite               /auth/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:4433;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+
+        }
+
+        location /packages/ {
+          try_files $uri =206;
+          auth_request          /auth/sessions/whoami;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+        } 
+        
         location /grafana/ {
           rewrite               /grafana/(.*) /$1 break;
           proxy_pass            http://{{ masterip }}:3000/;
@@ -100,7 +170,7 @@ http {
         }
 
         location /kibana/ {
-          auth_request          /so-auth/api/auth/;
+          auth_request          /auth/sessions/whoami;
           rewrite               /kibana/(.*) /$1 break;
           proxy_pass            http://{{ masterip }}:5601/;
           proxy_read_timeout    90;
@@ -112,6 +182,19 @@ http {
 
         }
 
+        location /nodered/ {
+          proxy_pass http://{{ masterip }}:1880/;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header Upgrade $http_upgrade;
+          proxy_set_header Connection "Upgrade";
+          proxy_set_header      Proxy "";
+
+        }
+        
         location /playbook/ {
           proxy_pass            http://{{ masterip }}:3200/playbook/;
           proxy_read_timeout    90;
@@ -125,7 +208,7 @@ http {
 
 
         location /navigator/ {
-          auth_request          /so-auth/api/auth/;
+          auth_request          /auth/sessions/whoami;
           proxy_pass            http://{{ masterip }}:4200/navigator/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
@@ -136,29 +219,21 @@ http {
 
         }
 
-        location /api/ {
-          proxy_pass            https://{{ masterip }}:8080/api/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_set_header      Upgrade $http_upgrade;
-          proxy_set_header      Connection "Upgrade";
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-
-        }
-
+      {%- if FLEET_NODE %}
         location /fleet/ {
-          proxy_pass            https://{{ masterip }}:8080/fleet/;
+          return 301 https://{{ FLEET_IP }}/fleet;
+        }
+      {%- else %}
+        location /fleet/ {
+          proxy_pass https://{{ masterip }}:8080;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
-
         }
+      {%- endif %}
 
         location /thehive/ {
           proxy_pass            http://{{ masterip }}:9000/thehive/;
@@ -195,22 +270,8 @@ http {
 
         }
 
-        location /sensoroni/ {
-          auth_request          /so-auth/api/auth/;
-          proxy_pass            http://{{ masterip }}:9822/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-          proxy_set_header      Upgrade $http_upgrade;
-          proxy_set_header      Connection "Upgrade";
-
-        }
-
-        location /kibana/app/sensoroni/ {
-          rewrite ^/kibana/app/sensoroni/(.*) /sensoroni/$1 permanent;
+        location /kibana/app/soc/ {
+          rewrite ^/kibana/app/soc/(.*) /soc/$1 permanent;
         }
 
         location /kibana/app/fleet/ {
@@ -231,23 +292,11 @@ http {
           proxy_set_header      Proxy "";
         }
 
-        location /so-auth/loginpage/ {
-            proxy_pass          http://{{ masterip }}:4242/;
-            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-        }
-
-        location /so-auth/api/ {
-            proxy_pass          http://{{ masterip }}:5656/;
-            proxy_set_header    X-Real-IP $remote_addr;
-            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header    X-Forwarded-Host $host;
-        }
-
         error_page 401 = @error401;
 
         location @error401 {
-          add_header    Set-Cookie "NSREDIRECT=http://{{ masterip }}$request_uri;Domain={{ masterip }};Path=/;Max-Age=60000";
-          return        302 http://{{ masterip }}/so-auth/loginpage/;
+          add_header    Set-Cookie "AUTH_REDIRECT=$request_uri;Path=/;Max-Age=14400";
+          return        302 /auth/self-service/browser/flows/login;
         }
 
         error_page 404 /404.html;

salt/common/nginx/nginx.conf.so-fleet

@@ -0,0 +1,98 @@
+{%- set MAINIP = salt['pillar.get']('node:mainip', '') %}
+# For more information on configuration, see:
+#   * Official English Documentation: http://nginx.org/en/docs/
+#   * Official Russian Documentation: http://nginx.org/ru/docs/
+
+user nginx;
+worker_processes auto;
+error_log /var/log/nginx/error.log;
+pid /run/nginx.pid;
+
+# Load dynamic modules. See /usr/share/nginx/README.dynamic.
+include /usr/share/nginx/modules/*.conf;
+
+events {
+    worker_connections 1024;
+}
+
+http {
+    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
+                      '$status $body_bytes_sent "$http_referer" '
+                      '"$http_user_agent" "$http_x_forwarded_for"';
+
+    access_log  /var/log/nginx/access.log  main;
+
+    sendfile            on;
+    tcp_nopush          on;
+    tcp_nodelay         on;
+    keepalive_timeout   65;
+    types_hash_max_size 2048;
+
+    include             /etc/nginx/mime.types;
+    default_type        application/octet-stream;
+
+    include /etc/nginx/conf.d/*.conf;
+
+    server {
+	      listen 80 default_server;
+	      server_name _;
+	      return 301 https://$host$request_uri;
+    }
+
+    server {
+        listen       8090 ssl http2 default_server;
+        server_name  _;
+        root         /opt/socore/html;
+        index        blank.html;
+
+        ssl_certificate "/etc/pki/nginx/server.crt";
+        ssl_certificate_key "/etc/pki/nginx/server.key";
+        ssl_session_cache shared:SSL:1m;
+        ssl_session_timeout  10m;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+
+        location ~ ^/kolide.agent.Api/(RequestEnrollment|RequestConfig|RequestQueries|PublishLogs|PublishResults|CheckHealth)$ {
+            grpc_pass  grpcs://{{ MAINIP }}:8080;
+            grpc_set_header Host $host;
+            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_buffering off;
+        }
+
+    }
+
+
+    server {
+        listen       443 ssl http2 default_server;
+        server_name  _;
+        root         /opt/socore/html/packages;
+        index        index.html;
+
+        ssl_certificate "/etc/pki/nginx/server.crt";
+        ssl_certificate_key "/etc/pki/nginx/server.key";
+        ssl_session_cache shared:SSL:1m;
+        ssl_session_timeout  10m;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+
+        location /fleet/ {
+          proxy_pass https://{{ MAINIP }}:8080;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+
+        }
+
+        error_page 404 /404.html;
+            location = /40x.html {
+        }
+
+        error_page 500 502 503 504 /50x.html;
+            location = /50x.html {
+        }
+    }
+
+}

salt/common/nginx/nginx.conf.so-master

@@ -1,9 +1,11 @@
 {%- set masterip = salt['pillar.get']('master:mainip', '') %}
+{%- set FLEET_MASTER = salt['pillar.get']('static:fleet_master') %}
+{%- set FLEET_NODE = salt['pillar.get']('static:fleet_node') %}
+{%- set FLEET_IP = salt['pillar.get']('static:fleet_ip', None) %}
 # For more information on configuration, see:
 #   * Official English Documentation: http://nginx.org/en/docs/
 #   * Official Russian Documentation: http://nginx.org/ru/docs/
 
-user nginx;
 worker_processes auto;
 error_log /var/log/nginx/error.log;
 pid /run/nginx.pid;
@@ -27,6 +29,7 @@ http {
     tcp_nodelay         on;
     keepalive_timeout   65;
     types_hash_max_size 2048;
+    client_max_body_size 1024M;
 
     include             /etc/nginx/mime.types;
     default_type        application/octet-stream;
@@ -58,11 +61,34 @@ http {
     #    }
     #}
     server {
-	      listen 80 default_server;
-	      server_name _;
-	      return 301 https://$host$request_uri;
+        listen 80 default_server;
+        server_name _;
+        return 301 https://$host$request_uri;
     }
 
+{% if FLEET_MASTER %}
+    server {
+        listen       8090 ssl http2 default_server;
+        server_name  _;
+        root         /opt/socore/html;
+        index        blank.html;
+
+        ssl_certificate "/etc/pki/nginx/server.crt";
+        ssl_certificate_key "/etc/pki/nginx/server.key";
+        ssl_session_cache shared:SSL:1m;
+        ssl_session_timeout  10m;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+
+        location ~ ^/kolide.agent.Api/(RequestEnrollment|RequestConfig|RequestQueries|PublishLogs|PublishResults|CheckHealth)$ {
+            grpc_pass  grpcs://{{ masterip }}:8080;
+            grpc_set_header Host $host;
+            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_buffering off;
+        }
+
+    }
+{% endif %}
 
 # Settings for a TLS enabled server.
 
@@ -83,13 +109,57 @@ http {
         # Load configuration files for the default server block.
         #include /etc/nginx/default.d/*.conf;
 
-        #location / {
-    #      try_files $uri $uri.html /index.html;
-    #    }
+        location ~* (^/login/|^/js/.*|^/css/.*|^/images/.*) {
+          proxy_pass            http://{{ masterip }}:9822;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+          proxy_set_header      Upgrade $http_upgrade;
+          proxy_set_header      Connection "Upgrade";
+        }
 
+        location / {
+          auth_request          /auth/sessions/whoami;
+          proxy_pass            http://{{ masterip }}:9822/;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+          proxy_set_header      Upgrade $http_upgrade;
+          proxy_set_header      Connection "Upgrade";
+        }
+
+        location ~ ^/auth/.*?(whoami|login|logout) {
+          rewrite               /auth/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:4433;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+
+        }
+
+        location /packages/ {
+          try_files $uri =206;
+          auth_request          /auth/sessions/whoami;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+        } 
+        
         location /grafana/ {
-          rewrite /grafana/(.*) /$1 break;
-          proxy_pass http://{{ masterip }}:3000/;
+          rewrite               /grafana/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:3000/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -100,9 +170,9 @@ http {
         }
 
         location /kibana/ {
-          auth_request          /so-auth/api/auth/;
-	        rewrite /kibana/(.*) /$1 break;
-          proxy_pass http://{{ masterip }}:5601/;
+          auth_request          /auth/sessions/whoami;
+          rewrite               /kibana/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:5601/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -112,8 +182,21 @@ http {
 
         }
 
-       location /playbook/ {
-          proxy_pass http://{{ masterip }}:3200/playbook/;
+        location /nodered/ {
+          proxy_pass http://{{ masterip }}:1880/;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header Upgrade $http_upgrade;
+          proxy_set_header Connection "Upgrade";
+          proxy_set_header      Proxy "";
+
+        }
+        
+        location /playbook/ {
+          proxy_pass            http://{{ masterip }}:3200/playbook/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -123,9 +206,10 @@ http {
 
         }
 
+
         location /navigator/ {
-          auth_request          /so-auth/api/auth/;
-          proxy_pass http://{{ masterip }}:4200/navigator/;
+          auth_request          /auth/sessions/whoami;
+          proxy_pass            http://{{ masterip }}:4200/navigator/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -135,37 +219,27 @@ http {
 
         }
 
-        location /api/ {
-          proxy_pass https://{{ masterip }}:8080/api/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_set_header      Upgrade $http_upgrade;
-          proxy_set_header      Connection "Upgrade";
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-
-        }
-
+      {%- if FLEET_NODE %}
         location /fleet/ {
-	        rewrite /fleet/(.*) /$1 break;
-          auth_request          /so-auth/api/auth/;
-          proxy_pass https://{{ masterip }}:8080/;
+          return 301 https://{{ FLEET_IP }}/fleet;
+        }
+      {%- else %}
+        location /fleet/ {
+          proxy_pass https://{{ masterip }}:8080;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
-
         }
+      {%- endif %}
 
         location /thehive/ {
-          proxy_pass http://{{ masterip }}:9000/thehive/;
+          proxy_pass            http://{{ masterip }}:9000/thehive/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
-          proxy_http_version 1.1; # this is essential for chunked responses to work
+          proxy_http_version    1.1; # this is essential for chunked responses to work
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -174,19 +248,19 @@ http {
         }
 
         location /cortex/ {
-          proxy_pass http://{{ masterip }}:9001/cortex/;
+          proxy_pass            http://{{ masterip }}:9001/cortex/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
-          proxy_http_version 1.1; # this is essential for chunked responses to work
+          proxy_http_version    1.1; # this is essential for chunked responses to work
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
 
         }
-
+        
         location /soctopus/ {
-          proxy_pass http://{{ masterip }}:7000/;
+          proxy_pass            http://{{ masterip }}:7000/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -196,22 +270,8 @@ http {
 
         }
 
-        location /sensoroni/ {
-          auth_request          /so-auth/api/auth/;
-          proxy_pass http://{{ masterip }}:9822/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-          proxy_set_header Upgrade $http_upgrade;
-          proxy_set_header Connection "Upgrade";
-
-        }
-
-        location /kibana/app/sensoroni/ {
-          rewrite ^/kibana/app/sensoroni/(.*) /sensoroni/$1 permanent;
+        location /kibana/app/soc/ {
+          rewrite ^/kibana/app/soc/(.*) /soc/$1 permanent;
         }
 
         location /kibana/app/fleet/ {
@@ -222,36 +282,21 @@ http {
           rewrite ^/kibana/app/soctopus/(.*) /soctopus/$1 permanent;
         }
 
-
         location /sensoroniagents/ {
-          proxy_pass http://{{ masterip }}:9822/;
+          proxy_pass            http://{{ masterip }}:9822/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
-
-        }
-
-
-        location /so-auth/loginpage/ {
-            proxy_pass          http://{{ masterip }}:4242/;
-            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-        }
-
-        location /so-auth/api/ {
-            proxy_pass          http://{{ masterip }}:5656/;
-            proxy_set_header    X-Real-IP $remote_addr;
-            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header    X-Forwarded-Host $host;
         }
 
         error_page 401 = @error401;
 
         location @error401 {
-          add_header    Set-Cookie "NSREDIRECT=http://{{ masterip }}$request_uri;Domain={{ masterip }};Path=/;Max-Age=60000";
-          return        302 http://{{ masterip }}/so-auth/loginpage/;
+          add_header    Set-Cookie "AUTH_REDIRECT=$request_uri;Path=/;Max-Age=14400";
+          return        302 /auth/self-service/browser/flows/login;
         }
 
         error_page 404 /404.html;

salt/common/nginx/nginx.conf.so-mastersearch

@@ -1,9 +1,11 @@
 {%- set masterip = salt['pillar.get']('master:mainip', '') %}
+{%- set FLEET_MASTER = salt['pillar.get']('static:fleet_master') %}
+{%- set FLEET_NODE = salt['pillar.get']('static:fleet_node') %}
+{%- set FLEET_IP = salt['pillar.get']('static:fleet_ip', None) %}
 # For more information on configuration, see:
 #   * Official English Documentation: http://nginx.org/en/docs/
 #   * Official Russian Documentation: http://nginx.org/ru/docs/
 
-user nginx;
 worker_processes auto;
 error_log /var/log/nginx/error.log;
 pid /run/nginx.pid;
@@ -27,6 +29,7 @@ http {
     tcp_nodelay         on;
     keepalive_timeout   65;
     types_hash_max_size 2048;
+    client_max_body_size 1024M;
 
     include             /etc/nginx/mime.types;
     default_type        application/octet-stream;
@@ -58,11 +61,34 @@ http {
     #    }
     #}
     server {
-	      listen 80 default_server;
-	      server_name _;
-	      return 301 https://$host$request_uri;
+        listen 80 default_server;
+        server_name _;
+        return 301 https://$host$request_uri;
     }
 
+{% if FLEET_MASTER %}
+    server {
+        listen       8090 ssl http2 default_server;
+        server_name  _;
+        root         /opt/socore/html;
+        index        blank.html;
+
+        ssl_certificate "/etc/pki/nginx/server.crt";
+        ssl_certificate_key "/etc/pki/nginx/server.key";
+        ssl_session_cache shared:SSL:1m;
+        ssl_session_timeout  10m;
+        ssl_ciphers HIGH:!aNULL:!MD5;
+        ssl_prefer_server_ciphers on;
+
+        location ~ ^/kolide.agent.Api/(RequestEnrollment|RequestConfig|RequestQueries|PublishLogs|PublishResults|CheckHealth)$ {
+            grpc_pass  grpcs://{{ masterip }}:8080;
+            grpc_set_header Host $host;
+            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+            proxy_buffering off;
+        }
+
+    }
+{% endif %}
 
 # Settings for a TLS enabled server.
 
@@ -83,13 +109,57 @@ http {
         # Load configuration files for the default server block.
         #include /etc/nginx/default.d/*.conf;
 
-        #location / {
-    #      try_files $uri $uri.html /index.html;
-    #    }
+        location ~* (^/login/|^/js/.*|^/css/.*|^/images/.*) {
+          proxy_pass            http://{{ masterip }}:9822;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+          proxy_set_header      Upgrade $http_upgrade;
+          proxy_set_header      Connection "Upgrade";
+        }
 
+        location / {
+          auth_request          /auth/sessions/whoami;
+          proxy_pass            http://{{ masterip }}:9822/;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+          proxy_set_header      Upgrade $http_upgrade;
+          proxy_set_header      Connection "Upgrade";
+        }
+
+        location ~ ^/auth/.*?(whoami|login|logout) {
+          rewrite               /auth/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:4433;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+
+        }
+
+        location /packages/ {
+          try_files $uri =206;
+          auth_request          /auth/sessions/whoami;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header      Proxy "";
+        } 
+        
         location /grafana/ {
-	        rewrite /grafana/(.*) /$1 break;
-          proxy_pass http://{{ masterip }}:3000/;
+          rewrite               /grafana/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:3000/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -100,9 +170,9 @@ http {
         }
 
         location /kibana/ {
-          auth_request          /so-auth/api/auth/;
-	        rewrite /kibana/(.*) /$1 break;
-          proxy_pass http://{{ masterip }}:5601/;
+          auth_request          /auth/sessions/whoami;
+          rewrite               /kibana/(.*) /$1 break;
+          proxy_pass            http://{{ masterip }}:5601/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -112,8 +182,21 @@ http {
 
         }
 
-       location /playbook/ {
-          proxy_pass http://{{ masterip }}:3200/playbook/;
+        location /nodered/ {
+          proxy_pass http://{{ masterip }}:1880/;
+          proxy_read_timeout    90;
+          proxy_connect_timeout 90;
+          proxy_set_header      Host $host;
+          proxy_set_header      X-Real-IP $remote_addr;
+          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
+          proxy_set_header Upgrade $http_upgrade;
+          proxy_set_header Connection "Upgrade";
+          proxy_set_header      Proxy "";
+
+        }
+        
+        location /playbook/ {
+          proxy_pass            http://{{ masterip }}:3200/playbook/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -123,9 +206,10 @@ http {
 
         }
 
+
         location /navigator/ {
-          auth_request          /so-auth/api/auth/;
-          proxy_pass http://{{ masterip }}:4200/navigator/;
+          auth_request          /auth/sessions/whoami;
+          proxy_pass            http://{{ masterip }}:4200/navigator/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -135,37 +219,27 @@ http {
 
         }
 
-        location /api/ {
-          proxy_pass https://{{ masterip }}:8080/api/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_set_header      Upgrade $http_upgrade;
-          proxy_set_header      Connection "Upgrade";
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-
-        }
-
+      {%- if FLEET_NODE %}
         location /fleet/ {
-	        rewrite /fleet/(.*) /$1 break;
-          auth_request          /so-auth/api/auth/;
-          proxy_pass https://{{ masterip }}:8080/;
+          return 301 https://{{ FLEET_IP }}/fleet;
+        }
+      {%- else %}
+        location /fleet/ {
+          proxy_pass https://{{ masterip }}:8080;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
-
         }
+      {%- endif %}
 
         location /thehive/ {
-          proxy_pass http://{{ masterip }}:9000/thehive/;
+          proxy_pass            http://{{ masterip }}:9000/thehive/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
-          proxy_http_version 1.1; # this is essential for chunked responses to work
+          proxy_http_version    1.1; # this is essential for chunked responses to work
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
@@ -174,31 +248,19 @@ http {
         }
 
         location /cortex/ {
-          proxy_pass http://{{ masterip }}:9001/cortex/;
+          proxy_pass            http://{{ masterip }}:9001/cortex/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
-          proxy_http_version 1.1; # this is essential for chunked responses to work
+          proxy_http_version    1.1; # this is essential for chunked responses to work
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
 
         }
-
-        location /cyberchef/ {
-          proxy_pass http://{{ masterip }}:9080/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_http_version 1.1; # this is essential for chunked responses to work
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-
-        }
-	
+        
         location /soctopus/ {
-          proxy_pass http://{{ masterip }}:7000/;
+          proxy_pass            http://{{ masterip }}:7000/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
@@ -208,22 +270,8 @@ http {
 
         }
 
-        location /sensoroni/ {
-          auth_request          /so-auth/api/auth/;
-          proxy_pass http://{{ masterip }}:9822/;
-          proxy_read_timeout    90;
-          proxy_connect_timeout 90;
-          proxy_set_header      Host $host;
-          proxy_set_header      X-Real-IP $remote_addr;
-          proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
-          proxy_set_header      Proxy "";
-          proxy_set_header Upgrade $http_upgrade;
-          proxy_set_header Connection "Upgrade";
-
-        }
-
-        location /kibana/app/sensoroni/ {
-          rewrite ^/kibana/app/sensoroni/(.*) /sensoroni/$1 permanent;
+        location /kibana/app/soc/ {
+          rewrite ^/kibana/app/soc/(.*) /soc/$1 permanent;
         }
 
         location /kibana/app/fleet/ {
@@ -234,36 +282,21 @@ http {
           rewrite ^/kibana/app/soctopus/(.*) /soctopus/$1 permanent;
         }
 
-
         location /sensoroniagents/ {
-          proxy_pass http://{{ masterip }}:9822/;
+          proxy_pass            http://{{ masterip }}:9822/;
           proxy_read_timeout    90;
           proxy_connect_timeout 90;
           proxy_set_header      Host $host;
           proxy_set_header      X-Real-IP $remote_addr;
           proxy_set_header      X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header      Proxy "";
-
-        }
-
-
-        location /so-auth/loginpage/ {
-            proxy_pass          http://{{ masterip }}:4242/;
-            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-        }
-
-        location /so-auth/api/ {
-            proxy_pass          http://{{ masterip }}:5656/;
-            proxy_set_header    X-Real-IP $remote_addr;
-            proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
-            proxy_set_header    X-Forwarded-Host $host;
         }
 
         error_page 401 = @error401;
 
         location @error401 {
-          add_header    Set-Cookie "NSREDIRECT=http://{{ masterip }}$request_uri;Domain={{ masterip }};Path=/;Max-Age=60000";
-          return        302 http://{{ masterip }}/so-auth/loginpage/;
+          add_header    Set-Cookie "AUTH_REDIRECT=$request_uri;Path=/;Max-Age=14400";
+          return        302 /auth/self-service/browser/flows/login;
         }
 
         error_page 404 /404.html;

salt/common/telegraf/etc/telegraf.conf

@@ -16,7 +16,7 @@
 {%- set MASTER = grains['master'] %}
 {% set NODEIP = salt['pillar.get']('node:mainip', '') %}
 {% set HELIX_API_KEY = salt['pillar.get']('fireeye:helix:api_key', '') %}
-
+{% set UNIQUEID = salt['pillar.get']('sensor:uniqueid', '') %}
 
 # Global tags can be specified here in key="value" format.
 [global_tags]
@@ -25,6 +25,9 @@
   ## Environment variables can be used as tags, and throughout the config file
   # user = "$USER"
 
+{% if grains['role'] == 'so-helix' %}
+  meta_cbid = "{{ UNIQUEID }}"
+{% endif %}
 
 # Configuration for telegraf agent
 [agent]
@@ -77,7 +80,7 @@
   logfile = "/var/log/telegraf/telegraf.log"
 
   ## Override default hostname, if empty use os.Hostname()
-  hostname = "{{ grains.id }}"
+  hostname = "{{ grains.host }}"
   ## If set to true, do no set the "host" tag in the telegraf agent.
   omit_hostname = false
 
@@ -495,10 +498,10 @@
 [[inputs.disk]]
   ## By default stats will be gathered for all mount points.
   ## Set mount_points will restrict the stats to only the specified mount points.
-  # mount_points = ["/"]
+  mount_points = ["/", "/host/nsm"]
 
   ## Ignore mount points by filesystem type.
-  ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]
+  #ignore_fs = ["tmpfs", "devtmpfs", "devfs", "overlay", "aufs", "squashfs"]
 
 
 # Read metrics about disk IO by device
@@ -2050,6 +2053,9 @@
 #   ## more about them here:
 #   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
 #   # data_format = "influx"
+[[inputs.socket_listener]]
+  service_address = "udp://:8094"
+  data_format = "influx"
 
 
 # # Statsd UDP/TCP Server

salt/common/telegraf/scripts/influxdbsize.sh

@@ -1,5 +1,5 @@
 #!/bin/bash
 
-INFLUXSIZE=$(du -s -B1 /host/nsm/influxdb | awk {'print $1'}
+INFLUXSIZE=$(du -s -B1 /host/nsm/influxdb | awk {'print $1'})
 
 echo "influxsize bytes=$INFLUXSIZE"

salt/common/tools/sbin/so-allow

@@ -56,7 +56,7 @@ if [ "$SKIP" -eq 0 ]; then
     echo ""
     echo "[a] - Analyst - ports 80/tcp and 443/tcp"
     echo "[b] - Logstash Beat - port 5044/tcp"
-    echo "[o] - Osquery endpoint - port 8080/tcp"
+    echo "[o] - Osquery endpoint - port 8090/tcp"
     echo "[w] - Wazuh endpoint - port 1514"
     echo ""
     echo "Please enter your selection (a - analyst, b - beats, o - osquery, w - wazuh):"

salt/common/tools/sbin/so-elastic-download

@@ -2,8 +2,6 @@
 MASTER=MASTER
 VERSION="HH1.1.4"
 TRUSTED_CONTAINERS=( \
-"so-auth-api:$VERSION" \
-"so-auth-ui:$VERSION" \
 "so-core:$VERSION" \
 "so-thehive-cortex:$VERSION" \
 "so-curator:$VERSION" \

salt/common/tools/sbin/so-elasticsearch-templates

@@ -0,0 +1,54 @@
+{% set MASTERIP = salt['pillar.get']('master:mainip', '') %}
+#!/bin/bash
+# Copyright 2014,2015,2016,2017,2018,2019 Security Onion Solutions, LLC
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+ELASTICSEARCH_HOST="{{ MASTERIP}}"
+ELASTICSEARCH_PORT=9200
+#ELASTICSEARCH_AUTH=""
+
+# Define a default directory to load pipelines from
+ELASTICSEARCH_TEMPLATES="/opt/so/saltstack/salt/logstash/pipelines/templates/so/"
+
+# Wait for ElasticSearch to initialize
+echo -n "Waiting for ElasticSearch..."
+COUNT=0
+ELASTICSEARCH_CONNECTED="no"
+while [[ "$COUNT" -le 240 ]]; do
+	curl --output /dev/null --silent --head --fail http://"$ELASTICSEARCH_HOST":"$ELASTICSEARCH_PORT"
+	if [ $? -eq 0 ]; then
+		ELASTICSEARCH_CONNECTED="yes"
+		echo "connected!"
+		break
+	else
+		((COUNT+=1))
+		sleep 1
+		echo -n "."
+	fi
+done
+if [ "$ELASTICSEARCH_CONNECTED" == "no" ]; then
+	echo
+	echo -e "Connection attempt timed out.  Unable to connect to ElasticSearch.  \nPlease try: \n  -checking log(s) in /var/log/elasticsearch/\n  -running 'sudo docker ps' \n  -running 'sudo so-elastic-restart'"
+	echo
+fi
+
+cd ${ELASTICSEARCH_TEMPLATES}
+
+
+echo "Loading templates..."
+for i in *; do TEMPLATE=$(echo $i | cut -d '-' -f2); echo "so-$TEMPLATE"; curl ${ELASTICSEARCH_AUTH} -s -XPUT http://${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}/_template/so-$TEMPLATE -H 'Content-Type: application/json' -d@$i 2>/dev/null; echo; done
+echo
+
+cd - >/dev/null

salt/common/tools/sbin/so-kibana-config-export

@@ -0,0 +1,6 @@
+
+#!/bin/bash
+KIBANA_HOST=10.66.166.141
+KSO_PORT=5601
+OUTFILE="saved_objects.json"
+curl -s -H 'kbn-xsrf: true' -H 'Content-Type: application/json' -XPOST $KIBANA_HOST:$KSO_PORT/api/saved_objects/_export -d '{ "type": "index-pattern", "type": "config", "type": "dashboard", "type": "query", "type": "search", "type": "url", "type": "visualization" }' -o $OUTFILE

salt/common/tools/sbin/so-nodered-restart

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-/usr/sbin/so-stop auth $1
+/usr/sbin/so-restart nodered $1

salt/common/tools/sbin/so-nodered-start

@@ -17,5 +17,5 @@
 
 . /usr/sbin/so-common
 
-/usr/sbin/so-restart auth $1
+/usr/sbin/so-start nodered $1
 

salt/common/tools/sbin/so-nodered-stop

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-/usr/sbin/so-start auth $1
+/usr/sbin/so-stop nodered $1

salt/common/tools/sbin/so-playbook-sync

@@ -17,4 +17,4 @@
 
 . /usr/sbin/so-common
 
-docker exec so-soctopus python3 playbook_play-sync.py
+docker exec so-soctopus python3 playbook_play-sync.py >> /opt/so/log/soctopus/so-playbook-sync.log 2>&1

salt/common/tools/sbin/so-restart

@@ -32,6 +32,5 @@ fi
 case $1 in
    "cortex") docker stop so-thehive-cortex so-thehive && docker rm so-thehive-cortex so-thehive && salt-call state.apply hive queue=True;;
    "steno") docker stop so-steno && docker rm so-steno && salt-call state.apply pcap queue=True;;
-   "auth") docker stop so-auth-api; docker stop so-auth-ui; salt-call state.apply auth queue=True;;
    *)  docker stop so-$1 ; docker rm so-$1 ; salt-call state.apply $1 queue=True;;
 esac

salt/common/tools/sbin/so-start

@@ -32,16 +32,5 @@ fi
 case $1 in
    "all") salt-call state.highstate queue=True;;
    "steno") if docker ps | grep -q so-$1; then printf "\n$1 is already running!\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply pcap queue=True; fi ;; 
-   "auth") 
-      if docker ps | grep -q so-auth-api; then
-         if docker ps | grep -q so-auth-ui; then
-            printf "\n$1 is already running!\n\n"
-         else
-            docker rm so-auth-api >/dev/null 2>&1; docker rm so-auth-ui >/dev/null 2>&1; salt-call state.apply $1 queue=True
-         fi
-      else
-         docker rm so-auth-api >/dev/null 2>&1; docker rm so-auth-ui >/dev/null 2>&1; salt-call state.apply $1 queue=True
-      fi
-      ;;
    *) if docker ps | grep -q so-$1; then printf "\n$1 is already running\n\n"; else docker rm so-$1 >/dev/null 2>&1 ; salt-call state.apply $1 queue=True; fi ;; 
 esac

salt/common/tools/sbin/so-status

@@ -26,6 +26,8 @@
     {%- set pillar_val = 'sensor' -%}
 {%- elif (salt['grains.get']('role') == 'so-eval') -%}
     {%- set pillar_val = 'eval' -%}
+{%- elif (salt['grains.get']('role') == 'so-fleet') -%}
+    {%- set pillar_val = 'fleet' -%}
 {%- elif (salt['grains.get']('role') == 'so-helix') -%}
     {%- set pillar_val = 'helix' -%}
 {%- elif (salt['grains.get']('role') == 'so-node') -%}
@@ -105,7 +107,7 @@ populate_container_lists() {
     systemctl is-active --quiet docker
 
     if [[ $? = 0 ]]; then
-        mapfile -t docker_raw_list < <(curl -s --unix-socket /var/run/docker.sock http:/containers/json?all=1 \
+        mapfile -t docker_raw_list < <(curl -s --unix-socket /var/run/docker.sock http:/v1.40/containers/json?all=1 \
             | jq -c '.[] | { Name: .Names[0], State: .State }' \
             | tr -d '/{"}')
     else

salt/common/tools/sbin/so-stop

@@ -24,7 +24,6 @@ printf "Stopping $1...\n"
 echo $banner
 
 case $1 in
-    "auth") docker stop so-auth-api; docker rm so-auth-api; docker stop so-auth-ui; docker rm so-auth-ui ;;
     *) docker stop so-$1 ; docker rm so-$1 ;;
 esac
 

salt/common/tools/sbin/so-user

@@ -0,0 +1,237 @@
+#!/bin/bash
+# Copyright 2020 Security Onion Solutions. All rights reserved.
+#
+# This program is distributed under the terms of version 2 of the
+# GNU General Public License.  See LICENSE for further details.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+got_root() {
+
+  # Make sure you are root
+  if [ "$(id -u)" -ne 0 ]; then
+          echo "This script must be run using sudo!"
+          exit 1
+  fi
+
+}
+
+# Make sure the user is root
+got_root
+
+if [[ $# < 1 || $# > 2 ]]; then
+  echo "Usage: $0 <list|add|update|delete|validate|valemail|valpass> [email]"
+  echo ""
+  echo "     list: Lists all user email addresses currently defined in the identity system"
+  echo "      add: Adds a new user to the identity system; requires 'email' parameter"
+  echo "   update: Updates a user's password; requires 'email' parameter"
+  echo "   delete: Deletes an existing user; requires 'email' parameter"
+  echo " validate: Validates that the given email address and password are acceptable for defining a new user; requires 'email' parameter"
+  echo " valemail: Validates that the given email address is acceptable for defining a new user; requires 'email' parameter"
+  echo "  valpass: Validates that a password is acceptable for defining a new user"
+  echo ""
+  echo " Note that the password can be piped into stdin to avoid prompting for it."
+  exit 1
+fi
+
+operation=$1
+email=$2
+
+kratosUrl=${KRATOS_URL:-http://127.0.0.1:4434}
+databasePath=${KRATOS_DB_PATH:-/opt/so/conf/kratos/db/db.sqlite}
+argon2Iterations=${ARGON2_ITERATIONS:-3}
+argon2Memory=${ARGON2_MEMORY:-14}
+argon2Parallelism=${ARGON2_PARALLELISM:-2}
+argon2HashSize=${ARGON2_HASH_SIZE:-32}
+
+function fail() {
+  msg=$1
+  echo "$1"
+  exit 1
+}
+
+function require() {
+  cmd=$1
+  which "$1" 2>&1 > /dev/null
+  [[ $? != 0 ]] && fail "This script requires the following command be installed: ${cmd}"
+}
+
+# Verify this environment is capable of running this script
+function verifyEnvironment() {
+  require "argon2"
+  require "jq"
+  require "curl"
+  require "openssl"
+  require "sqlite3"
+  [[ ! -f $databasePath ]] && fail "Unable to find database file; specify path via KRATOS_DB_PATH environment variable"
+  response=$(curl -Ss ${kratosUrl}/)
+  [[ "$response" != "404 page not found" ]] && fail "Unable to communicate with Kratos; specify URL via KRATOS_URL environment variable"
+}
+
+function findIdByEmail() {
+  email=$1
+
+  response=$(curl -Ss ${kratosUrl}/identities)
+  identityId=$(echo "${response}" | jq ".[] | select(.addresses[0].value == \"$email\") | .id")
+  echo $identityId
+}
+
+function validatePassword() {
+  password=$1
+
+  len=$(expr length "$password")
+  if [[ $len -lt 6 ]]; then
+    echo "Password does not meet the minimum requirements"
+    exit 2
+  fi
+}
+
+function validateEmail() {
+  email=$1
+  # (?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
+  if [[ ! "$email" =~ ^[[:alnum:]._%+-]+@[[:alnum:].-]+\.[[:alpha:]]{2,}$ ]]; then
+    echo "Email address is invalid"
+    exit 3
+  fi
+}
+
+function updatePassword() {
+  identityId=$1
+  
+  # Read password from stdin (show prompt only if no stdin was piped in)
+  test -t 0
+  if [[ $? == 0 ]]; then
+    echo "Enter new password:"
+  fi
+  read -s password
+
+  validatePassword "$password"
+
+  if [[ -n $identityId ]]; then
+    # Generate password hash
+    salt=$(openssl rand -hex 8)
+    passwordHash=$(echo "${password}" | argon2 ${salt} -id -t $argon2Iterations -m $argon2Memory -p $argon2Parallelism -l $argon2HashSize -e)
+
+    # Update DB with new hash
+    echo "update identity_credentials set config=CAST('{\"hashed_password\":\"${passwordHash}\"}' as BLOB) where identity_id=${identityId};" | sqlite3 "$databasePath"
+    [[ $? != 0 ]] && fail "Unable to update password"
+  fi
+}
+
+function listUsers() {
+  response=$(curl -Ss ${kratosUrl}/identities)
+  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
+
+  echo "${response}" | jq -r ".[] | .addresses[0].value" | sort
+}
+
+function createUser() {
+  email=$1
+
+  now=$(date -u +%FT%TZ)
+  addUserJson=$(cat <<EOF
+{
+  "addresses": [
+    {
+      "expires_at": "2099-01-31T12:00:00Z",
+      "value": "${email}",
+      "verified": true,
+      "verified_at": "${now}",
+      "via": "so-add-user"
+    }
+  ],
+  "traits": {"email":"${email}"},
+  "traits_schema_id": "default"
+}
+EOF
+  )
+  
+  response=$(curl -Ss ${kratosUrl}/identities -d "$addUserJson")
+  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
+
+  identityId=$(echo "${response}" | jq ".id")
+  if [[ ${identityId} == "null" ]]; then
+    code=$(echo "${response}" | jq ".error.code")
+    [[ "${code}" == "409" ]] && fail "User already exists"
+
+    reason=$(echo "${response}" | jq ".error.message")
+    [[ $? == 0 ]] && fail "Unable to add user: ${reason}"
+  fi
+
+  updatePassword $identityId
+}
+
+function updateUser() {
+  email=$1
+
+  identityId=$(findIdByEmail "$email")
+  [[ ${identityId} == "" ]] && fail "User not found"
+
+  updatePassword $identityId 
+}
+
+function deleteUser() {
+  email=$1
+
+  identityId=$(findIdByEmail "$email")
+  [[ ${identityId} == "" ]] && fail "User not found"
+
+  response=$(curl -Ss -XDELETE "${kratosUrl}/identities/$identityId")
+  [[ $? != 0 ]] && fail "Unable to communicate with Kratos"
+}
+
+case "${operation}" in
+  "add")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+
+    validateEmail "$email"
+    createUser "$email"
+    echo "Successfully added new user"
+    ;;
+
+  "list")
+    verifyEnvironment
+    listUsers
+    ;;
+
+  "update")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+
+    updateUser "$email"
+    echo "Successfully updated user"
+    ;;
+
+  "delete")
+    verifyEnvironment
+    [[ "$email" == "" ]] && fail "Email address must be provided"
+
+    deleteUser "$email"
+    echo "Successfully deleted user"  
+    ;;
+
+  "validate")
+    validateEmail "$email"
+    updatePassword
+    echo "Email and password are acceptable"
+    ;;
+
+  "valemail")
+    validateEmail "$email"
+    echo "Email is acceptable"
+    ;;
+
+  "valpass")
+    updatePassword
+    echo "Password is acceptable"
+    ;;
+
+  *)
+    fail "Unsupported operation: $operation"
+    ;;
+esac
+
+exit 0

salt/common/tools/sbin/so-user-add

@@ -1,17 +1,2 @@
 #!/bin/bash
-USERNAME=$1
-
-# Make sure a username is provided
-[ $# -eq 0 ] && { echo "Usage: $0 username"; exit 1; }
-
-# If the file is there already lets create it otherwise add the user
-if [ ! -f /opt/so/conf/nginx/.htpasswd ]; then
-
-  # Create the password file
-  htpasswd -c /opt/so/conf/nginx/.htpasswd $USERNAME
-
-else
-
-  htpasswd /opt/so/conf/nginx/.htpasswd $USERNAME
-
-fi
+so-user add $*

salt/common/tools/sbin/soup

@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Copyright 2014,2015,2016,2017,2018,2019,2020 Security Onion Solutions, LLC
+
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License as published by
+#    the Free Software Foundation, either version 3 of the License, or
+#    (at your option) any later version.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+clone_to_tmp() {
+
+  # TODO Need to add a air gap option
+  # Make a temp location for the files
+  rm -rf /tmp/soup
+  mkdir -p /tmp/soup
+  cd /tmp/soup
+  #git clone -b dev https://github.com/Security-Onion-Solutions/securityonion-saltstack.git
+  git clone https://github.com/Security-Onion-Solutions/securityonion-saltstack.git
+
+}
+
+# Prompt the user that this requires internets
+
+clone_to_tmp
+cd /tmp/soup/securityonion-saltstack/update
+chmod +x soup
+./soup
+
+

salt/curator/files/bin/so-curator-closed-delete

@@ -34,8 +34,6 @@
 #fi
 
 # Avoid starting multiple instances
-if pgrep -f "so-curator-closed-delete-delete" >/dev/null; then
-	echo "Script is already running."
-else
+if ! pgrep -f "so-curator-closed-delete-delete" >/dev/null; then
 	/usr/sbin/so-curator-closed-delete-delete
 fi

salt/curator/init.sls

@@ -1,4 +1,4 @@
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.1.4') %}
+{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.1') %}
 {% set MASTER = salt['grains.get']('master') %}
 {% if grains['role'] == 'so-node' or grains['role'] == 'so-eval' %}
 # Curator
@@ -87,8 +87,9 @@ curdel:
     - group: 939
     - mode: 755
 
-/usr/sbin/so-curator-closed-delete:
+so-curatorcloseddeletecron:
  cron.present:
+   - name: /usr/sbin/so-curator-closed-delete
    - user: root
    - minute: '*'
    - hour: '*'
@@ -96,8 +97,9 @@ curdel:
    - month: '*'
    - dayweek: '*'
 
-/usr/sbin/so-curator-close:
+so-curatorclosecron:
  cron.present:
+   - name: /usr/sbin/so-curator-close
    - user: root
    - minute: '*'
    - hour: '*'
@@ -105,8 +107,9 @@ curdel:
    - month: '*'
    - dayweek: '*'
 
-/usr/sbin/so-curator-delete:
+so-curatordeletecron:
  cron.present:
+   - name: /usr/sbin/so-curator-delete
    - user: root
    - minute: '*'
    - hour: '*'

salt/elastalert/files/modules/so/playbook-es.py

@@ -0,0 +1,23 @@
+# -*- coding: utf-8 -*-
+
+from datetime import date   
+import requests,json
+from elastalert.alerts import Alerter
+
+class PlaybookESAlerter(Alerter):
+    """
+    Use matched data to create alerts in elasticsearch
+    """
+
+    required_options = set(['play_title','play_url','sigma_level','elasticsearch_host'])
+
+    def alert(self, matches):
+       for match in matches:
+            headers = {"Content-Type": "application/json"}
+            payload = {"play_title": self.rule['play_title'],"play_url": self.rule['play_url'],"sigma_level": self.rule['sigma_level'],"data": match}
+            today = str(date.today())
+            url = f"http://{self.rule['elasticsearch_host']}/playbook-alerts-{today}/_doc/"
+            requests.post(url, data=json.dumps(payload), headers=headers, verify=False)
+                            
+    def get_info(self):
+        return {'type': 'PlaybookESAlerter'} 

salt/elastalert/files/rules/so/nids2hive.yaml

@@ -8,21 +8,20 @@ es_host: {{es}}
 es_port: 9200
 name: NIDS-Alert
 type: frequency
-index: "*:logstash-ids*"
+index: "so-ids-*"
 num_events: 1
 timeframe:
     minutes: 10
 buffer_time:
     minutes: 10
 allow_buffer_time_overlap: true
-query_key: ["alert", "ips"]
+query_key: ["rule.signature_id"]
 realert:
     days: 1
-
 filter:
 - query:
    query_string:
-      query: "event_type: ids AND NOT tags: _jsonparsefailure"
+      query: "event.module: suricata"
 
 alert: modules.so.thehive.TheHiveAlerter
 
@@ -35,16 +34,16 @@ hive_proxies:
   https: ''
 
 hive_alert_config:
-  title: '{match[alert]}'
+  title: '{match[rule][name]}'
   type: 'NIDS'
   source: 'SecurityOnion'
-  description: "`NIDS Dashboard:` \n\n <https://{{es}}/kibana/app/kibana#/dashboard/ed6f7e20-e060-11e9-8f0c-2ddbf5ed9290?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:'*:logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'sid:{match[sid]}')),sort:!('@timestamp',desc))> \n\n `IPs: `{match[source_ip]}:{match[source_port]} --> {match[destination_ip]}:{match[destination_port]} \n\n `Signature:` {match[rule_signature]}"
+  description: "`NIDS Dashboard:` \n\n <https://{{es}}/kibana/app/kibana#/dashboard/ed6f7e20-e060-11e9-8f0c-2ddbf5ed9290?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:'*:logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'sid:')),sort:!('@timestamp',desc))> \n\n `IPs: `{match[source][ip]}:{match[source][port]} --> {match[destination][ip]}:{match[destination][port]} \n\n `Signature:`{match[rule][rule]}"
   severity: 2
-  tags: ['{match[sid]}','{match[source_ip]}','{match[destination_ip]}']
+  tags: ['{match[rule][signature_id]}','{match[source][ip]}','{match[destination][ip]}']
   tlp: 3
   status: 'New'
   follow: True
 
 hive_observable_data_mapping:
-  - ip: '{match[source_ip]}'
-  - ip: '{match[destination_ip]}'
+  - ip: '{match[source][ip]}'
+  - ip: '{match[destination][ip]}'

salt/elastalert/init.sls

@@ -12,7 +12,7 @@
 #
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
-{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.1.4') %}
+{% set VERSION = salt['pillar.get']('static:soversion', 'HH1.2.1') %}
 {% set MASTER = salt['grains.get']('master') %}
 {% if grains['role'] == 'so-master' %}
 

salt/elasticsearch/files/ingest/bro_common

@@ -1,9 +0,0 @@
-{
-  "description" : "bro_common",
-  "processors" : [
-    { "rename":		{ "field": "@timestamp",	"target_field": "timestamp",	"ignore_missing": true					} },
-    { "date":		{ "field": "message2.ts",	"target_field": "@timestamp",	"formats": ["ISO8601", "UNIX"], "ignore_failure": true	} },
-    { "remove":		{ "field": "message2.ts",	"ignore_failure": true									} },
-    { "pipeline":	{ "name": "common" 													} }
-  ]
-}

salt/elasticsearch/files/ingest/bro_conn

@@ -1,48 +0,0 @@
-{
-  "description" : "bro_conn",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.proto", 		"target_field": "protocol",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.service", 		"target_field": "service",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.duration", 	"target_field": "duration",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_bytes", 	"target_field": "original_bytes",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_bytes", 	"target_field": "respond_bytes",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.conn_state", 	"target_field": "connection_state",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.local_orig", 	"target_field": "local_orig",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.local_resp", 	"target_field": "local_respond",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.missed_bytes", 	"target_field": "missed_bytes",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.history", 		"target_field": "history",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_pkts", 	"target_field": "original_packets",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_ip_bytes", 	"target_field": "original_ip_bytes",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_pkts", 	"target_field": "respond_packets",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_ip_bytes", 	"target_field": "respond_ip_bytes",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.tunnel_parents",	"target_field": "tunnel_parents",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_cc", 		"target_field": "original_country_code","ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_cc", 		"target_field": "respond_country_code",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sensorname", 	"target_field": "sensor_name",		"ignore_missing": true 	} },
-    { "script":		{ "lang": "painless", "source":	"ctx.total_bytes = (ctx.original_bytes + ctx.respond_bytes)", "ignore_failure": true } },
-    { "set": { "if": "ctx.connection_state == 'S0'", 	"field": "connection_state_description", "value": "Connection attempt seen, no reply" } },
-    { "set": { "if": "ctx.connection_state == 'S1'", 	"field": "connection_state_description", "value": "Connection established, not terminated" } },
-    { "set": { "if": "ctx.connection_state == 'S2'", 	"field": "connection_state_description", "value": "Connection established and close attempt by originator seen (but no reply from responder)" } },
-    { "set": { "if": "ctx.connection_state == 'S3'", 	"field": "connection_state_description", "value": "Connection established and close attempt by responder seen (but no reply from originator)" } },
-    { "set": { "if": "ctx.connection_state == 'SF'", 	"field": "connection_state_description", "value": "Normal SYN/FIN completion" } },
-    { "set": { "if": "ctx.connection_state == 'REJ'", 	"field": "connection_state_description", "value": "Connection attempt rejected" } },
-    { "set": { "if": "ctx.connection_state == 'RSTO'", 	"field": "connection_state_description", "value": "Connection established, originator aborted (sent a RST)" } },
-    { "set": { "if": "ctx.connection_state == 'RSTR'", 	"field": "connection_state_description", "value": "Established, responder aborted" } },
-    { "set": { "if": "ctx.connection_state == 'RSTOS0'","field": "connection_state_description", "value": "Originator sent a SYN followed by a RST, we never saw a SYN-ACK from the responder" } },
-    { "set": { "if": "ctx.connection_state == 'RSTRH'", "field": "connection_state_description", "value": "Responder sent a SYN ACK followed by a RST, we never saw a SYN from the (purported) originator" } },
-    { "set": { "if": "ctx.connection_state == 'SH'", 	"field": "connection_state_description", "value": "Originator sent a SYN followed by a FIN, we never saw a SYN ACK from the responder (hence the connection was 'half' open)" } },
-    { "set": { "if": "ctx.connection_state == 'SHR'", 	"field": "connection_state_description", "value": "Responder sent a SYN ACK followed by a FIN, we never saw a SYN from the originator" } },
-    { "set": { "if": "ctx.connection_state == 'OTH'", 	"field": "connection_state_description", "value": "No SYN seen, just midstream traffic (a 'partial connection' that was not later closed)" } },
-    { "pipeline": 	{ "name": "bro_common" } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_dce_rpc

@@ -1,20 +0,0 @@
-{
-  "description" : "bro_dce_rpc",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rtt", 		"target_field": "rtt",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.named_pipe",	"target_field": "named_pipe",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.endpoint", 	"target_field": "endpoint",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.operation", 	"target_field": "operation",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_dnp3

@@ -1,19 +0,0 @@
-{
-  "description" : "bro_dnp3",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fc_request",	"target_field": "fc_request",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fc_reply", 	"target_field": "fc_reply",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.iin", 		"target_field": "iin",			"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_dns

@@ -1,35 +0,0 @@
-{
-  "description" : "bro_dns",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.proto", 		"target_field": "protocol",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.trans_id",		"target_field": "transaction_id",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rtt",	 	"target_field": "rtt",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.query", 		"target_field": "query",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.qclass", 		"target_field": "query_class",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.qclass_name", 	"target_field": "query_class_name",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.qtype", 		"target_field": "query_type",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.qtype_name", 	"target_field": "query_type_name",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rcode",	 	"target_field": "rcode",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rcode_name", 	"target_field": "rcode_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.AA", 		"target_field": "aa",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.TC", 		"target_field": "tc",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.RD", 		"target_field": "rd",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.RA", 		"target_field": "ra",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.Z",		"target_field": "z",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.answers", 		"target_field": "answers",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.TTLs", 		"target_field": "ttls",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rejected", 	"target_field": "rejected",		"ignore_missing": true 	} },
-    { "script": 	{ "lang": "painless", "source": "ctx.query_length = ctx.query.length()",	"ignore_failure": true  } },
-    { "pipeline":       { "name": "bro_common"											} }
-  ]
-}

salt/elasticsearch/files/ingest/bro_files

@@ -1,32 +0,0 @@
-{
-  "description" : "bro_files",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.fuid", 	 	"target_field": "fuid",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.tx_hosts", 	"target_field": "file_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rx_hosts.0",	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "remove": 	{ "field": "message2.rx_hosts",							"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.conn_uids", 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "remove": 	{ "field": "source",								"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.source", 		"target_field": "source",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.depth",	 	"target_field": "depth",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.analyzers", 	"target_field": "analyzer",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mime_type", 	"target_field": "mimetype",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.filename", 	"target_field": "file_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.duration", 	"target_field": "duration",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.local_orig",	"target_field": "local_orig",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.is_orig", 		"target_field": "is_orig",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.seen_bytes", 	"target_field": "seen_bytes",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.total_bytes", 	"target_field": "total_bytes",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.missing_bytes", 	"target_field": "missing_bytes",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.overflow_bytes",	"target_field": "overflow_bytes",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.timedout",		"target_field": "timed_out",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.parent_fuid",	"target_field": "parent_fuid",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.md5",		"target_field": "md5",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sha1",		"target_field": "sha1",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.extracted",	"target_field": "extracted",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.extracted_cutoff",	"target_field": "extracted_cutoff",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.extracted_size",	"target_field": "extracted_size",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_ftp

@@ -1,33 +0,0 @@
-{
-  "description" : "bro_http",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.user",		"target_field": "username",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.password",		"target_field": "password",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.command", 		"target_field": "ftp_command",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.arg", 		"target_field": "ftp_argument",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mime_type", 	"target_field": "mimetype",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.file_size", 	"target_field": "file_size",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.reply_code", 	"target_field": "reply_code",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.reply_msg", 	"target_field": "reply_message",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "data_channel.passive", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.data_channel.passive","target_field": "data_channel_passive",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "data_channel.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.data_channel.orig_h","target_field": "data_channel_source_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "data_channel.resp_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.data_channel.resp_h","target_field": "data_channel_destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "data_channel.resp_p", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.data_channel.resp_p","target_field": "data_channel_destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fuid",		"target_field": "fuid",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_http

@@ -1,42 +0,0 @@
-{
-  "description" : "bro_http",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.trans_depth",	"target_field": "trans_depth",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.method", 		"target_field": "method",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.host", 		"target_field": "virtual_host",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uri", 		"target_field": "uri",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.referrer", 	"target_field": "referrer",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.version", 		"target_field": "version",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.user_agent", 	"target_field": "useragent",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_body_len", "target_field": "request_body_length",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.response_body_len","target_field": "response_body_length",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.status_code",	"target_field": "status_code",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.status_msg", 	"target_field": "status_message",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.info_code", 	"target_field": "info_code",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.info_msg", 	"target_field": "info_message",		"ignore_missing": true 	} },
-    { "remove":		{ "field": "message2.tags",		"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.username", 	"target_field": "user",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.password", 	"target_field": "password",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.proxied", 		"target_field": "proxied",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_fuids",	"target_field": "orig_fuids",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_filenames",	"target_field": "orig_filenames",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.orig_mime_types",	"target_field": "orig_mime_types",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_fuids",	"target_field": "resp_fuids",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_filenames",	"target_field": "resp_filenames",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resp_mime_types",	"target_field": "resp_mime_types",	"ignore_missing": true 	} },
-    { "script":	{ "lang": "painless", "source": "ctx.uri_length = ctx.uri.length()", 			"ignore_failure": true 	} },
-    { "script":	{ "lang": "painless", "source": "ctx.useragent_length = ctx.useragent.length()",	"ignore_failure": true 	} },
-    { "script":	{ "lang": "painless", "source": "ctx.virtual_host_length = ctx.virtual_host.length()", 	"ignore_failure": true	} },
-    { "pipeline":       { "name": "bro_common"											} }
-  ]
-}

salt/elasticsearch/files/ingest/bro_intel

@@ -1,29 +0,0 @@
-{
-  "description" : "bro_intel",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "seen.indicator", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.seen.indicator", 	"target_field": "indicator",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "seen.indicator_type", 	"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.seen.indicator_type", 	"target_field": "indicator_type",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "seen.where", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.seen.where", 	"target_field": "seen_where",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "seen.node", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.seen.node", 	"target_field": "seen_node",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.matched", 		"target_field": "matched",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sources", 		"target_field": "sources",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fuid", 		"target_field": "fuid",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.file_mime_type", 	"target_field": "mimetype",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.file_desc", 	"target_field": "file_description",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_irc

@@ -1,25 +0,0 @@
-{
-  "description" : "bro_irc",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.nick", 		"target_field": "nick",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.user", 		"target_field": "irc_username",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.command", 		"target_field": "irc_command",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.value", 		"target_field": "value",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.addl", 		"target_field": "additional_info",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dcc_file_name", 	"target_field": "dcc_file_name",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dcc_file_size", 	"target_field": "dcc_file_size",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dcc_mime_type", 	"target_field": "dcc_mime_type",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fuid", 		"target_field": "fuid",			"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_kerberos

@@ -1,30 +0,0 @@
-{
-  "description" : "bro_kerberos",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_type", 	"target_field": "request_type",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client", 		"target_field": "client",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.service", 		"target_field": "service",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.success", 		"target_field": "kerberos_success",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.error_msg", 	"target_field": "error_message",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.from", 		"target_field": "valid_from",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.till", 		"target_field": "valid_till",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cipher", 		"target_field": "cipher",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.forwardable", 	"target_field": "forwardable",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.renewable", 	"target_field": "renewable",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_cert_subject", 	"target_field": "client_certificate_subject",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_cert_fuid", 	"target_field": "client_certificate_fuid",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_cert_subject", 	"target_field": "server_certificate_subject",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_cert_fuid", 	"target_field": "server_certificate_fuid",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_modbus

@@ -1,18 +0,0 @@
-{
-  "description" : "bro_modbus",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.func", 		"target_field": "function",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.exception", 	"target_field": "exception",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_mysql

@@ -1,21 +0,0 @@
-{
-  "description" : "bro_mysql",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cmd", 		"target_field": "mysql_command",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.arg", 		"target_field": "mysql_argument",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.success", 		"target_field": "mysql_success",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rows", 		"target_field": "rows",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.response",		"target_field": "response",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_notice

@@ -1,36 +0,0 @@
-{
-  "description" : "bro_notice",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "remove":		{ "field": "message2.dst",		"ignore_failure": true						} },
-    { "remove":		{ "field": "message2.src",		"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fuid", 		"target_field": "fuid",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mime", 		"target_field": "file_mime_type",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.desc", 		"target_field": "file_description",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.proto", 		"target_field": "protocol",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.note", 		"target_field": "note",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.msg", 		"target_field": "msg",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sub", 		"target_field": "sub_msg",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.p", 		"target_field": "p",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.n", 		"target_field": "n",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.peer_descr",	"target_field": "peer_description",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.actions", 		"target_field": "action",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.suppress_for", 	"target_field": "suppress_for",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.dropped", 		"target_field": "dropped",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_country_code",	"target_field": "destination_country_code",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_region", 	"target_field": "destination_region",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_city", 	"target_field": "destination_city",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_latitude", 	"target_field": "destination_latitude",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_longitude", 	"target_field": "destination_longitude",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_ntlm

@@ -1,24 +0,0 @@
-{
-  "description" : "bro_ntlm",
-  "processors" : [
-    { "json":		{ "field": "message",				"target_field": "message2",			"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 		"target_field": "uid",				"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 			"path": "message2", 				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 		"target_field": "source_ip",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 			"path": "message2",				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 		"target_field": "source_port",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 			"path": "message2",				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 		"target_field": "destination_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 			"path": "message2",				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 		"target_field": "destination_port",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.hostname",			"target_field": "hostname",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.domainname",		"target_field": "domain_name",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.success", 			"target_field": "ntlm_success",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.status", 			"target_field": "status",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.username",			"target_field": "username",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_dns_computer_name",	"target_field": "server_dns_computer_name", 	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_nb_computer_name",	"target_field": "server_nb_computer_name", 	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_tree_name",		"target_field": "server_tree_name",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   		} }
-  ]
-}

salt/elasticsearch/files/ingest/bro_pe

@@ -1,23 +0,0 @@
-{
-  "description" : "bro_pe",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.id", 	 	"target_field": "fuid",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.machine",		"target_field": "machine",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.compile_ts",	"target_field": "compile_ts",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.os", 		"target_field": "os",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.subsystem",	"target_field": "subsystem",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.is_exe", 		"target_field": "is_exe",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.is_64bit",		"target_field": "is_64bit",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uses_aslr", 	"target_field": "uses_aslr",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uses_dep", 	"target_field": "uses_dep",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uses_code_integrity","target_field": "uses_code_integrity","ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uses_seh",		"target_field": "uses_seh",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.has_import_table", "target_field": "has_import_table",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.has_export_table", "target_field": "has_export_table",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.has_cert_table", 	"target_field": "has_cert_table",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.has_debug_data", 	"target_field": "has_debug_data",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.section_names", 	"target_field": "section_names",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_radius

@@ -1,25 +0,0 @@
-{
-  "description" : "bro_radius",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.username", 	"target_field": "username",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mac", 		"target_field": "mac",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.framed_addr", 	"target_field": "framed_addr",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.remote_ip", 	"target_field": "remote_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.connect_info", 	"target_field": "connect_info",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.reply_msg", 	"target_field": "reply_message",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.result", 		"target_field": "result",		"ignore_missing": true 	} },
-    { "remove":		{ "field": "message2.ttl",		"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.logged", 		"target_field": "logged",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_rdp

@@ -1,31 +0,0 @@
-{
-  "description" : "bro_rdp",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cookie", 		"target_field": "cookie",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.result", 		"target_field": "result",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.security_protocol","target_field": "security_protocol",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.keyboard_layout", 	"target_field": "keyboard_layout",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_build", 	"target_field": "client_build",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_name", 	"target_field": "client_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_dig_product_id", 	"target_field": "client_digital_product_id",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.desktop_width", 	"target_field": "desktop_width",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.desktop_height", 	"target_field": "desktop_height",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.requested_color_depth", 	"target_field": "requested_color_depth",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cert_type", 	"target_field": "certificate_type",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cert_count", 	"target_field": "certificate_count",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cert_permanent", 	"target_field": "certificate_permanent","ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.encryption_level",	"target_field": "encryption_level",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.encryption_method","target_field": "encryption_method",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_rfb

@@ -1,26 +0,0 @@
-{
-  "description" : "bro_rfb",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_major_version",	"target_field": "client_major_version",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_minor_version",	"target_field": "client_minor_version",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_major_version",	"target_field": "server_major_version",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_minor_version",	"target_field": "server_minor_version",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.authentication_method",	"target_field": "authentication_method","ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.auth", 			"target_field": "auth",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.share_flag", 	"target_field": "share_flag",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.desktop_name", 	"target_field": "desktop_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.width", 		"target_field": "width",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.height", 		"target_field": "height",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_signatures

@@ -1,22 +0,0 @@
-{
-  "description" : "bro_signatures",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.note", 		"target_field": "note",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sig_id", 		"target_field": "signature_id",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.event_msg", 	"target_field": "event_message",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sub_msg", 		"target_field": "sub_message",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sig_count", 	"target_field": "signature_count",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.host_count", 	"target_field": "host_count",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_sip

@@ -1,37 +0,0 @@
-{
-  "description" : "bro_sip",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.trans_depth", 	"target_field": "trans_depth",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.method", 		"target_field": "method",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uri", 		"target_field": "uri",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.date", 		"target_field": "date",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_from", 	"target_field": "request_from",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_to", 	"target_field": "request_to",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.response_from", 	"target_field": "response_from",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.response_to", 	"target_field": "response_to",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.reply_to", 	"target_field": "reply_to",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.call_id", 		"target_field": "call_id",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.seq", 		"target_field": "seq",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.subject", 		"target_field": "subject",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_path", 	"target_field": "request_path",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.response_path", 	"target_field": "response_path",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.user_agent", 	"target_field": "user_agent",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.status_code", 	"target_field": "status_code",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.status_msg",	"target_field": "status_msg",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.warning", 		"target_field": "warning",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_body_len",	"target_field": "request_body_length",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.response_body_len","target_field": "response_body_length",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.content_type", 	"target_field": "content_type",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_smb_files

@@ -1,31 +0,0 @@
-{
-  "description" : "bro_smb_files",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fuid", 		"target_field": "fuid",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.action", 		"target_field": "action",		"ignore_missing": true 	} },
-    { "remove":		{ "field": "path",			"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.path", 		"target_field": "path",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.name", 		"target_field": "name",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.size", 		"target_field": "size",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.prev_name", 	"target_field": "prev_name",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "times.modified", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.times.modified", 	"target_field": "times_modified",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "times.accessed", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.times.accessed", 	"target_field": "times_accessed",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "times.created", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.times.created", 	"target_field": "times_created",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "times.changed", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.times.changed", 	"target_field": "times_changed",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_smb_mapping

@@ -1,21 +0,0 @@
-{
-  "description" : "bro_smb_files",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "remove":		{ "field": "path",			"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.path", 		"target_field": "path",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.service", 		"target_field": "service",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.native_file_system",	"target_field": "native_file_system",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.share_type",	"target_field": "share_type",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_smtp

@@ -1,38 +0,0 @@
-{
-  "description" : "bro_smtp",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "remove":		{ "field": "path",			"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.trans_depth", 	"target_field": "trans_depth",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.helo", 		"target_field": "helo",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mailfrom", 	"target_field": "mail_from",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rcptto", 		"target_field": "recipient_to",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.date", 		"target_field": "mail_date",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.from", 		"target_field": "from",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.to", 		"target_field": "to",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cc", 		"target_field": "cc",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.reply_to", 	"target_field": "reply_to",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.msg_id", 		"target_field": "message_id",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.in_reply_to", 	"target_field": "in_reply_to",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.subject", 		"target_field": "subject",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.x_originating_ip",	"target_field": "x_originating_ip",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.first_received", 	"target_field": "first_received",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.second_received",	"target_field": "second_received",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.last_reply",	"target_field": "last_reply",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.path", 		"target_field": "path",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.user_agent", 	"target_field": "useragent",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.tls", 		"target_field": "tls",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.fuids", 		"target_field": "fuids",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.is_webmail",	"target_field": "is_webmail",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_snmp

@@ -1,25 +0,0 @@
-{
-  "description" : "bro_snmp",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.duration",		"target_field": "duration",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.version", 		"target_field": "version",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.community", 	"target_field": "community",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.get_requests", 	"target_field": "get_requests",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.get_bulk_requests","target_field": "get_bulk_requests",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.get_responses",	"target_field": "get_responses",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.set_requests", 	"target_field": "set_requests",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.display_string", 	"target_field": "display_string",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.up_since", 	"target_field": "up_since",		"ignore_missing": true 	} },
-    { "pipeline":	{ "name": "bro_common"											} }
-  ]
-}

salt/elasticsearch/files/ingest/bro_socks

@@ -1,28 +0,0 @@
-{
-  "description" : "bro_socks",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.version", 		"target_field": "version",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.user", 		"target_field": "user",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.password", 	"target_field": "password",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.status", 		"target_field": "status",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_host", 	"target_field": "request_host",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "request.name", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.request.name", 	"target_field": "request_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.request_p", 	"target_field": "request_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "bound.host", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.bound.host", 	"target_field": "bound_host",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.bound_name", 	"target_field": "bound_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.bound_p", 		"target_field": "bound_port",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_ssh

@@ -1,40 +0,0 @@
-{
-  "description" : "bro_conn",
-  "processors" : [
-    { "json":		{ "field": "message",				"target_field": "message2",			"ignore_failure": true	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 			"path": "message2", 				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 		"target_field": "source_ip",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 			"path": "message2",				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 		"target_field": "source_port",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 			"path": "message2",				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 		"target_field": "destination_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 			"path": "message2",				"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 		"target_field": "destination_port",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.version", 			"target_field": "version",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.uid", 			"target_field": "uid",				"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.hassh", 			"target_field": "hassh",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.auth_success",		"target_field": "authentication_success",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.auth_attempts",		"target_field": "authentication_attempts",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.direction", 		"target_field": "direction",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client", 			"target_field": "client",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server", 			"target_field": "server",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cipher_alg", 		"target_field": "cipher_algorithm",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.compression_alg", 		"target_field": "compression_algorithm",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cshka", 			"target_field": "client_host_key_algorithms",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.host_key_alg", 		"target_field": "host_key_algorithm",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.hasshAlgorithms", 		"target_field": "hassh_algorithms",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.hasshServer", 		"target_field": "hassh_server",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.hasshVersion", 		"target_field": "hassh_version",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.kex_alg", 			"target_field": "kex_algorithm",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mac_alg", 			"target_field": "mac_algorithm",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.sshka", 			"target_field": "server_host_key_algorithms",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.host_key", 		"target_field": "host_key",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_region", 	"target_field": "destination_region",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_city", 	"target_field": "destination_city",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_latitude", 	"target_field": "destination_latitude",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_longitude", 	"target_field": "destination_longitude",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.destination_country_code", "target_field": "destination_country_code",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.hasshServerAlgorithms", 	"target_field": "hassh_server_algorithms",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   		} }
-  ]
-}

salt/elasticsearch/files/ingest/bro_ssl

@@ -1,33 +0,0 @@
-{
-  "description" : "bro_ssl",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.version", 		"target_field": "version",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cipher", 		"target_field": "cipher",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.curve", 		"target_field": "curve",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_name", 	"target_field": "server_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.resumed", 		"target_field": "resumed",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.last_alert", 	"target_field": "last_alert",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.next_protocol", 	"target_field": "next_protocol",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.established", 	"target_field": "established",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.cert_chain_fuids", 	"target_field": "certificate_chain_fuids",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_cert_chain_fuids",	"target_field": "client_certificate_chain_fuids",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.subject", 		"target_field": "certificate_subject",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.issuer", 		"target_field": "certificate_issuer",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_subject", 	"target_field": "client_subject",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_issuer", 	"target_field": "client_issuer",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.validation_status","target_field": "validation_status",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.ja3", 		"target_field": "ja3",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.ja3s", 		"target_field": "ja3s",			"ignore_missing": true 	} },
-    { "pipeline":       { "name":  "bro_common_ssl"                                                                             } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_syslog

@@ -1,21 +0,0 @@
-{
-  "description" : "bro_syslog",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.proto", 		"target_field": "protocol",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.facility", 	"target_field": "facility",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.severity", 	"target_field": "severity",		"ignore_missing": true 	} },
-    { "remove":		{ "field": "message",			"ignore_failure": true						} },
-    { "rename": 	{ "field": "message2.message", 		"target_field": "message",		"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_tunnel

@@ -1,7 +0,0 @@
-{
-  "description" : "bro_tunnel",
-  "processors" : [
-    { "set":		{ "field": "event_type", "value": "bro_tunnels"								} },
-    { "pipeline":       { "name": "bro_tunnels"                                                                                 } }
-  ]
-}

salt/elasticsearch/files/ingest/bro_weird

@@ -1,20 +0,0 @@
-{
-  "description" : "bro_weird",
-  "processors" : [
-    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uid", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_h", 		"path": "message2", 			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_h", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.orig_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.orig_p", 	"target_field": "source_port",		"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_h", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_h", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "dot_expander": 	{ "field": "id.resp_p", 		"path": "message2",			"ignore_failure": true 	} },
-    { "rename": 	{ "field": "message2.id.resp_p", 	"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.name", 		"target_field": "name",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.addl", 		"target_field": "additional_info",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.notice", 		"target_field": "notice",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.peer", 		"target_field": "peer",			"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
-  ]
-}

salt/elasticsearch/files/ingest/common

@@ -1,17 +1,10 @@
 {
   "description" : "common",
   "processors" : [
-    { 
-	"rename": {
-		"field": "type",
-		"target_field": "event_type",
-		"ignore_missing": true
-	}
-    },
     { 	
 	"geoip":  { 
-		"field": "destination_ip",
-		"target_field": "destination_geo",
+		"field": "destination.ip",
+		"target_field": "geo",
 		"database_file": "GeoLite2-City.mmdb",
 		"ignore_missing": true, 
 		"properties": ["ip", "country_iso_code", "country_name", "continent_name", "region_iso_code", "region_name", "city_name", "timezone", "location"]
@@ -19,8 +12,8 @@
     },
     { 
 	"geoip":  {
-		"field": "source_ip",
-		"target_field": "source_geo",
+		"field": "source.ip",
+		"target_field": "geo",
 		"database_file": "GeoLite2-City.mmdb",
 		"ignore_missing": true,
 		"properties": ["ip", "country_iso_code", "country_name", "continent_name", "region_iso_code", "region_name", "city_name", "timezone", "location"]
@@ -41,11 +34,15 @@
 	        "ignore_failure": true,
         	"index_name_format": "yyyy.MM.dd"
         }
-    },
+    },    
+    { "rename":         { "field": "module",              "target_field": "event.module",                  "ignore_missing": true  } },
+    { "rename":         { "field": "dataset",              "target_field": "event.dataset",                  "ignore_missing": true  } },
+    { "rename":         { "field": "category",              "target_field": "event.category",                  "ignore_missing": true  } },
+    { "rename":         { "field": "message2.community_id",              "target_field": "network.community_id",                  "ignore_missing": true  } },
     { 
 	"remove": {
-		"field": "index_name_prefix",
-		"ignore_failure": true
+		"field": [ "index_name_prefix", "message2"],
+		"ignore_failure": false
 	}
     }
   ]

salt/elasticsearch/files/ingest/osquery.query_result

@@ -0,0 +1,55 @@
+{
+  "description" : "osquery",
+  "processors" : [
+    { "json":        { "field": "message",                                            "target_field": "message2",             "ignore_failure": true  } },
+    { "gsub":        { "field": "message2.columns.data",                              "pattern": "\\\\xC2\\\\xAE",             "replacement": "", "ignore_missing": true  } },
+    { "json":        { "field": "message2.columns.data",                              "target_field": "message2.columns.winlog",             "ignore_failure": true  } },
+    {
+      "script": {
+        "lang": "painless",
+        "source": "def dict = ['result': new HashMap()]; for (entry in ctx['message2'].entrySet()) { dict['result'][entry.getKey()] = entry.getValue(); } ctx['osquery'] = dict; "
+      }
+    },
+    { "rename":      { "field": "osquery.result.hostIdentifier",                            "target_field": "osquery.result.host_identifier",              "ignore_missing": true  } },
+    { "rename":      { "field": "osquery.result.calendarTime",                              "target_field": "osquery.result.calendar_time",              "ignore_missing": true  } },
+    { "rename":      { "field": "osquery.result.unixTime",                                  "target_field": "osquery.result.unix_time",                "ignore_missing": true  } },    
+    { "json":        { "field": "message",                                                  "target_field": "message3",             "ignore_failure": true  } },
+    { "gsub":        { "field": "message3.columns.data",                                    "pattern": "\\\\xC2\\\\xAE",             "replacement": "", "ignore_missing": true  } },
+    { "json":        { "field": "message3.columns.data",                                    "target_field": "message3.columns.winlog",             "ignore_failure": true  } },
+    { "rename":      { "field": "message3.columns.username",                                "target_field": "user.name",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.uid",                                     "target_field": "user.uid",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.gid",                                     "target_field": "user.gid",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.shell",                                   "target_field": "user.shell",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.cmdline",                                 "target_field": "process.command_line",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.pid",                                     "target_field": "process.pid",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.parent",                                  "target_field": "process.ppid",              "ignore_missing": true  } },
+    { "rename":      { "field": "message3.columns.cwd",                                     "target_field": "process.working_directory",              "ignore_missing": true  } },
+    { "rename": 	   { "field": "message3.columns.eventid", 			                          "target_field": "event.code",		"ignore_missing": true 	} },
+    { "set":         { "if": "ctx.message3.columns.data != null",   "field": "dataset", "value": "wel-{{message3.columns.source}}", "override": true }  },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.SubjectUserName", 			  "target_field": "user.name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.destinationHostname", 	  "target_field": "destination.hostname",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.destinationIp", 		      "target_field": "destination.ip",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.destinationPort", 	      "target_field": "destination.port",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.Image", 			            "target_field": "process.executable",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.ProcessID", 			        "target_field": "process.pid",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.ProcessGuid", 			      "target_field": "process.entity_id",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.CommandLine", 			      "target_field": "process.command_line",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.CurrentDirectory", 			"target_field": "process.working_directory",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.Description", 			      "target_field": "process.pe.description",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.Product", 			          "target_field": "process.pe.product",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.OriginalFileName", 			"target_field": "process.pe.original_file_name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.FileVersion", 			      "target_field": "process.pe.file_version",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentCommandLine", 			"target_field": "process.parent.command_line",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentImage", 			      "target_field": "process.parent.executable",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentProcessGuid", 			"target_field": "process.parent.entity_id",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.ParentProcessId", 			  "target_field": "process.ppid",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.User", 			            "target_field": "user.name",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.parentImage", 		        "target_field": "parent_image_path",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.sourceHostname", 	      "target_field": "source.hostname",	"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.sourceIp", 		          "target_field": "source_ip",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.sourcePort", 		        "target_field": "source.port",		"ignore_missing": true 	} },
+    { "rename": 	   { "field": "message3.columns.winlog.EventData.targetFilename",		      "target_field": "file.target",	"ignore_missing": true 	} },
+    { "remove":      { "field": [ "message3"],                                              "ignore_failure": false } },
+    { "pipeline":    { "name": "common" } }
+  ]
+}

salt/elasticsearch/files/ingest/ossec

@@ -6,48 +6,47 @@
     { "rename": 	{ "field": "message2.data", 				"target_field": "data",			"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.decoder", 				"target_field": "decoder",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.full_log", 			"target_field": "full_log",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.id", 				"target_field": "id",			"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.id", 				"target_field": "log.id.id",			"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.location", 			"target_field": "location",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.manager", 				"target_field": "manager",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.predecoder", 			"target_field": "predecoder",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.timestamp", 			"target_field": "timestamp",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.rule", 				"target_field": "wazuh-rule",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.rule", 				"target_field": "rule",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "data.command", 				"target_field": "command",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.dstip", 				"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.dstport", 				"target_field": "destination_port",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.dstuser", 				"target_field": "escalated_user",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.srcip", 				"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.srcuser", 				"target_field": "username",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.destinationHostname", 	"target_field": "destination_hostname",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.destinationIp", 		"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.destinationPort", 	"target_field": "destination_port",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.dstip", 				"target_field": "destination.ip",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.dstport", 				"target_field": "destination.port",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.dstuser", 				"target_field": "user.escalated",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.srcip", 				"target_field": "source.ip",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.srcuser", 				"target_field": "source.user",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.eventdata.destinationHostname", 	"target_field": "destination.hostname",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.eventdata.destinationIp", 		"target_field": "destination.ip",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.eventdata.destinationPort", 	"target_field": "destination.port",	"ignore_missing": true 	} },
     { "rename": 	{ "field": "data.win.eventdata.image", 			"target_field": "image_path",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "data.win.eventdata.parentImage", 		"target_field": "parent_image_path",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.sourceHostname", 	"target_field": "source_hostname",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.eventdata.sourceHostname", 	"target_field": "source.hostname",	"ignore_missing": true 	} },
     { "rename": 	{ "field": "data.win.eventdata.sourceIp", 		"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.sourcePort", 		"target_field": "source_port",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.targetFilename",		"target_field": "target_filename",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.eventdata.user", 			"target_field": "username",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "data.win.system.eventID", 			"target_field": "event_id",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "predecoder.program_name", 			"target_field": "process",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "wazuh-rule.level", 				"target_field": "alert_level",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "wazuh-rule.description", 			"target_field": "description",		"ignore_missing": true 	} },
-    { "set": 		{ "if": "ctx.alert_level == 1", 		"field": "classification", "value": "None" 				} },
-    { "set": 		{ "if": "ctx.alert_level == 2", 		"field": "classification", "value": "System low priority notification" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 3", 		"field": "classification", "value": "Successful/authorized event" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 4", 		"field": "classification", "value": "System low priority error" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 5", 		"field": "classification", "value": "User generated error" 		} },
-    { "set": 		{ "if": "ctx.alert_level == 6", 		"field": "classification", "value": "Low relevance attack" 		} },
-    { "set": 		{ "if": "ctx.alert_level == 7", 		"field": "classification", "value": "\"Bad word\" matching" 		} },
-    { "set": 		{ "if": "ctx.alert_level == 8", 		"field": "classification", "value": "First time seen" 			} },
-    { "set": 		{ "if": "ctx.alert_level == 9", 		"field": "classification", "value": "Error from invalid source" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 10", 		"field": "classification", "value": "Multiple user generated errors" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 11", 		"field": "classification", "value": "Integrity checking warning" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 12", 		"field": "classification", "value": "High importance event" 		} },
-    { "set": 		{ "if": "ctx.alert_level == 13", 		"field": "classification", "value": "Unusal error (high importance)" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 14", 		"field": "classification", "value": "High importance security event" 	} },
-    { "set": 		{ "if": "ctx.alert_level == 15", 		"field": "classification", "value": "Severe attack" 			} },
-    { "append": 	{ "if": "ctx.alert_level != null",		"field": "tags", "value": ["alert"] 					} },
+    { "rename": 	{ "field": "data.win.eventdata.sourcePort", 		"target_field": "source.port",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.eventdata.targetFilename",		"target_field": "file.target",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.eventdata.user", 			"target_field": "user.name",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "data.win.system.eventID", 			"target_field": "event.code",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "predecoder.program_name", 			"target_field": "process.name",		"ignore_missing": true 	} },
+    { "set": 		{ "if": "ctx.rule.level == 1", 		"field": "rule.category", "value": "None" 				} },
+    { "set": 		{ "if": "ctx.rule.level == 2", 		"field": "rule.category", "value": "System low priority notification" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 3", 		"field": "rule.category", "value": "Successful/authorized event" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 4", 		"field": "rule.category", "value": "System low priority error" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 5", 		"field": "rule.category", "value": "User generated error" 		} },
+    { "set": 		{ "if": "ctx.rule.level == 6", 		"field": "rule.category", "value": "Low relevance attack" 		} },
+    { "set": 		{ "if": "ctx.rule.level == 7", 		"field": "rule.category", "value": "\"Bad word\" matching" 		} },
+    { "set": 		{ "if": "ctx.rule.level == 8", 		"field": "rule.category", "value": "First time seen" 			} },
+    { "set": 		{ "if": "ctx.rule.level == 9", 		"field": "rule.category", "value": "Error from invalid source" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 10", 		"field": "rule.category", "value": "Multiple user generated errors" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 11", 		"field": "rule.category", "value": "Integrity checking warning" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 12", 		"field": "rule.category", "value": "High importance event" 		} },
+    { "set": 		{ "if": "ctx.rule.level == 13", 		"field": "rule.category", "value": "Unusal error (high importance)" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 14", 		"field": "rule.category", "value": "High importance security event" 	} },
+    { "set": 		{ "if": "ctx.rule.level == 15", 		"field": "rule.category", "value": "Severe attack" 			} },
+    { "append": 	{ "if": "ctx.rule.level != null",		"field": "tags", "value": ["alert"] 					} },
+    { "remove": { "field": [ "host", "predecoder", "decoder" ], "ignore_missing": true, "ignore_failure": false } },
     { "pipeline":	{ "name": "common" } }
   ]
 }

salt/elasticsearch/files/ingest/ossec.alert

@@ -0,0 +1,56 @@
+{
+  "description" : "ossec",
+  "processors" : [
+    { "json":           { "field": "message",                                   "target_field": "message2",             "ignore_failure": true  } },
+    { "remove": { "field": [ "agent" ], "ignore_missing": true, "ignore_failure": false } },
+    { "rename":         { "field": "message2.agent",                            "target_field": "agent",                "ignore_missing": true  } },
+    { "rename":         { "field": "message2.data",                             "target_field": "data",                 "ignore_missing": true  } },
+    { "rename":         { "field": "message2.decoder",                          "target_field": "decoder",              "ignore_missing": true  } },
+    { "rename":         { "field": "message2.full_log",                         "target_field": "log.full",             "ignore_missing": true  } },
+    { "rename":         { "field": "message2.id",                               "target_field": "log.id.id",                    "ignore_missing": true  } },
+    { "rename":         { "field": "message2.location",                         "target_field": "log.location",         "ignore_missing": true  } },
+    { "rename":         { "field": "message2.manager",                          "target_field": "manager",              "ignore_missing": true  } },
+    { "rename":         { "field": "message2.predecoder",                       "target_field": "predecoder",           "ignore_missing": true  } },
+    { "rename":         { "field": "message2.timestamp",                        "target_field": "event.timestamp",              "ignore_missing": true  } },
+    { "rename":         { "field": "message2.previous_log",                        "target_field": "log.previous_log",            "ignore_missing": true  } },
+    { "rename":         { "field": "message2.previous_output",                        "target_field": "log.previous_output",            "ignore_missing": true  } },
+    { "rename":         { "field": "message2.rule",                             "target_field": "rule",         "ignore_missing": true  } },
+    { "rename":         { "field": "message2.syscheck",                         "target_field": "host.syscheck",        "ignore_missing": true  } },
+    { "rename":         { "field": "data.command",                              "target_field": "process.command_line",         "ignore_missing": true  } },
+    { "rename":         { "field": "data.dstip",                                "target_field": "destination.ip",       "ignore_missing": true  } },
+    { "rename":         { "field": "data.dstport",                              "target_field": "destination.port",     "ignore_missing": true  } },
+    { "rename":         { "field": "data.dstuser",                              "target_field": "user.escalated",       "ignore_missing": true  } },
+    { "rename":         { "field": "data.srcip",                                "target_field": "source.ip",            "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.destinationHostname",    "target_field": "destination.hostname", "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.destinationIp",          "target_field": "destination.ip",       "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.destinationPort",        "target_field": "destination.port",     "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.image",                  "target_field": "image_path",           "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.parentImage",            "target_field": "parent_image_path",    "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.sourceHostname",         "target_field": "source.hostname",      "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.sourceIp",               "target_field": "source_ip",            "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.sourcePort",             "target_field": "source.port",          "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.targetFilename",         "target_field": "file.target",  "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.eventdata.user",                   "target_field": "user.name",            "ignore_missing": true  } },
+    { "rename":         { "field": "data.win.system.eventID",                   "target_field": "event.code",           "ignore_missing": true  } },
+    { "rename":         { "field": "predecoder.program_name",                   "target_field": "process.name",         "ignore_missing": true  } },
+    { "rename":         { "field": "rule.description",                   "target_field": "rule.name",         "ignore_missing": true  } },
+    { "set":            { "if": "ctx.rule.level == 1",          "field": "rule.category", "value": "None"                               } },
+    { "set":            { "if": "ctx.rule.level == 2",          "field": "rule.category", "value": "System low priority notification"   } },
+    { "set":            { "if": "ctx.rule.level == 3",          "field": "rule.category", "value": "Successful/authorized event"        } },
+    { "set":            { "if": "ctx.rule.level == 4",          "field": "rule.category", "value": "System low priority error"  } },
+    { "set":            { "if": "ctx.rule.level == 5",          "field": "rule.category", "value": "User generated error"               } },
+    { "set":            { "if": "ctx.rule.level == 6",          "field": "rule.category", "value": "Low relevance attack"               } },
+    { "set":            { "if": "ctx.rule.level == 7",          "field": "rule.category", "value": "\"Bad word\" matching"              } },
+    { "set":            { "if": "ctx.rule.level == 8",          "field": "rule.category", "value": "First time seen"                    } },
+    { "set":            { "if": "ctx.rule.level == 9",          "field": "rule.category", "value": "Error from invalid source"  } },
+    { "set":            { "if": "ctx.rule.level == 10",                 "field": "rule.category", "value": "Multiple user generated errors"     } },
+    { "set":            { "if": "ctx.rule.level == 11",                 "field": "rule.category", "value": "Integrity checking warning"         } },
+    { "set":            { "if": "ctx.rule.level == 12",                 "field": "rule.category", "value": "High importance event"              } },
+    { "set":            { "if": "ctx.rule.level == 13",                 "field": "rule.category", "value": "Unusal error (high importance)"     } },
+    { "set":            { "if": "ctx.rule.level == 14",                 "field": "rule.category", "value": "High importance security event"     } },
+    { "set":            { "if": "ctx.rule.level == 15",                 "field": "rule.category", "value": "Severe attack"                      } },
+    { "append":         { "if": "ctx.rule.level != null",               "field": "tags", "value": ["alert"]                                     } },
+    { "remove": { "field": [ "predecoder", "decoder" ], "ignore_missing": true, "ignore_failure": false } },
+    { "pipeline":       { "name": "common" } }
+  ]
+}

salt/elasticsearch/files/ingest/strelka

@@ -0,0 +1,12 @@
+{
+  "description" : "strelka",
+  "processors" : [
+    { "json":           { "field": "message",                        "target_field": "message2",                   "ignore_failure": true  } },
+    { "rename":         { "field": "message2.file",        "target_field": "file",          "ignore_missing": true  } },
+    { "rename":         { "field": "message2.scan",        "target_field": "scan",          "ignore_missing": true  } },
+    { "rename":         { "field": "message2.request",        "target_field": "request",          "ignore_missing": true  } },
+    { "rename":         { "field": "scan.hash",        "target_field": "file.hash",          "ignore_missing": true  } },
+    { "remove":         { "field": ["host", "path", "agent"],                                         "ignore_missing": true  } },
+    { "pipeline":       { "name": "common"                                                                                   } }
+  ]
+}

salt/elasticsearch/files/ingest/strelka.file

@@ -0,0 +1,12 @@
+{
+  "description" : "strelka",
+  "processors" : [
+    { "json":           { "field": "message",                        "target_field": "message2",                   "ignore_failure": true  } },
+    { "rename":         { "field": "message2.file",        "target_field": "file",          "ignore_missing": true  } },
+    { "rename":         { "field": "message2.scan",        "target_field": "scan",          "ignore_missing": true  } },
+    { "rename":         { "field": "message2.request",        "target_field": "request",          "ignore_missing": true  } },
+    { "rename":         { "field": "scan.hash",        "target_field": "hash",          "ignore_missing": true  } },
+    { "remove":         { "field": ["host", "path"],                                         "ignore_missing": true  } },
+    { "pipeline":       { "name": "common"                                                                                   } }
+  ]
+}

salt/elasticsearch/files/ingest/suricata.alert

@@ -0,0 +1,11 @@
+{
+  "description" : "suricata.alert",
+  "processors" : [
+    { "json":           { "field": "message",                   "target_field": "message2",             "ignore_failure": true  } },
+    { "rename":{ "field": "message2.comunity_id",       "target_field": "network.comunity_id",  "ignore_failure": true  } },
+    { "rename":{ "field": "message2.alert",	"target_field": "rule",	"ignore_failure": true 	} },
+    { "rename":{ "field": "rule.signature",     "target_field": "rule.name", "ignore_failure": true  } },
+    { "rename":{ "field": "rule.ref",     "target_field": "rule.version", "ignore_failure": true  } },
+    { "pipeline": { "name": "suricata.common" } }
+  ]
+}

salt/elasticsearch/files/ingest/suricata.common

@@ -0,0 +1,15 @@
+{
+  "description" : "suricata.common",
+  "processors" : [
+    { "rename":{ "field": "message2.proto",		"target_field": "network.transport",		"ignore_failure": true 	} },
+    { "rename":{ "field": "message2.flow_id",		"target_field": "event.id",		"ignore_failure": true 	} },
+    { "rename":{ "field": "message2.src_ip",       "target_field": "source.ip",  "ignore_failure": true  } },
+    { "rename":{ "field": "message2.src_port",       "target_field": "source.port",  "ignore_failure": true  } },
+    { "rename":{ "field": "message2.dest_ip",       "target_field": "destination.ip",  "ignore_failure": true  } },
+    { "rename":{ "field": "message2.dest_port",       "target_field": "destination.port",  "ignore_failure": true  } },
+    { "rename":         { "field": "message2.community_id",        "target_field": "network.community_id",     "ignore_missing": true  } },
+    { "set":         { "field": "observer.name",        "value": "{{agent.name}}"  } },
+    { "remove":         { "field": ["agent"],     "ignore_failure": true                                                                  } },
+    { "pipeline": { "name": "common" } }
+  ]
+}

salt/elasticsearch/files/ingest/zeek.common

@@ -0,0 +1,23 @@
+{
+  "description" : "zeek.common",
+  "processors" : [
+    { "rename":		{ "field": "@timestamp",	"target_field": "ingest.timestamp",	"ignore_missing": true					} },
+    { "rename":         { "field": "message2.uid",              "target_field": "log.id.uid",                  "ignore_missing": true  } },
+    { "dot_expander":   { "field": "id.orig_h",                 "path": "message2",                     "ignore_failure": true  } },
+    { "dot_expander":   { "field": "id.orig_p",                 "path": "message2",                     "ignore_failure": true  } },
+    { "dot_expander":   { "field": "id.resp_h",                 "path": "message2",                     "ignore_failure": true  } },
+    { "dot_expander":   { "field": "id.resp_p",                 "path": "message2",                     "ignore_failure": true  } },
+    { "rename":         { "field": "message2.id.orig_h",        "target_field": "source.ip",            "ignore_missing": true  } },
+    { "rename":         { "field": "message2.id.orig_p",        "target_field": "source.port",          "ignore_missing": true  } },
+    { "rename":         { "field": "message2.id.resp_h",        "target_field": "destination.ip",       "ignore_missing": true  } },
+    { "rename":         { "field": "message2.id.resp_p",        "target_field": "destination.port",     "ignore_missing": true  } },
+    { "set":         { "field": "client.ip",        "value": "{{source.ip}}"           } },
+    { "set":         { "if": "ctx.source.port != null", "field": "client.port",        "value": "{{source.port}}"       } },
+    { "set":         { "field": "server.ip",        "value": "{{destination.ip}}"  } },
+    { "set":         { "if": "ctx.destination.port != null", "field": "server.port",        "value": "{{destination.port}}"  } },
+    { "set":         { "field": "observer.name",        "value": "{{agent.name}}"  } },
+    { "date":		{ "field": "message2.ts",	"target_field": "@timestamp",	"formats": ["ISO8601", "UNIX"], "ignore_failure": true	} },
+    { "remove":		{ "field": ["agent"],	"ignore_failure": true									} },
+    { "pipeline":	{ "name": "common" 													} }
+  ]
+}

salt/elasticsearch/files/ingest/zeek.common_ssl

@@ -1,5 +1,5 @@
 {
-  "description" : "bro_common_ssl",
+  "description" : "zeek.common_ssl",
   "processors" : [
     {
   	"kv": {
@@ -51,7 +51,7 @@
     { 	"script":{ "lang": "painless", "source": "ctx.server_name_length = ctx.server_name.length()", "ignore_failure": true } },
     { 
 	"pipeline": { 
-		"name": "bro_common"
+		"name": "zeek.common"
 	}
     }
   ]

salt/elasticsearch/files/ingest/zeek.conn

@@ -0,0 +1,26 @@
+{
+  "description" : "zeek.conn",
+  "processors" : [
+    { "remove":         { "field": ["host"],     "ignore_failure": true                                                              } },
+    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
+    { "rename": 	{ "field": "message2.proto", 		"target_field": "network.transport",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.service", 		"target_field": "network.protocol",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.duration", 	"target_field": "event.duration",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.orig_bytes", 	"target_field": "client.bytes",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.resp_bytes", 	"target_field": "server.bytes",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.conn_state", 	"target_field": "connection.state",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.local_orig", 	"target_field": "connection.local.originator",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.local_resp", 	"target_field": "connection.local.responder",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.missed_bytes", 	"target_field": "connection.bytes.missed",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.history", 		"target_field": "connection.history",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.orig_pkts", 	"target_field": "client.packets",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.orig_ip_bytes", 	"target_field": "client.ip_bytes",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.resp_pkts", 	"target_field": "server.packets",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.resp_ip_bytes", 	"target_field": "server.ip_bytes",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.tunnel_parents",	"target_field": "connection.tunnel_parents",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.orig_cc", 		"target_field": "client.country_code","ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.resp_cc", 		"target_field": "server.country_code",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.sensorname", 	"target_field": "observer.name",		"ignore_missing": true 	} },
+    { "pipeline": 	{ "name": "zeek.common" } }
+  ]
+}

salt/elasticsearch/files/ingest/zeek.dce_rpc

@@ -0,0 +1,12 @@
+{
+  "description" : "zeek.dce_rpc",
+  "processors" : [
+    { "remove":         { "field": ["host"],     "ignore_failure": true                                                                  } },
+    { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
+    { "rename": 	{ "field": "message2.rtt", 		"target_field": "event.duration",			"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.named_pipe",	"target_field": "named_pipe",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.endpoint", 	"target_field": "endpoint",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.operation", 	"target_field": "operation",		"ignore_missing": true 	} },
+    { "pipeline":       { "name": "zeek.common"                                                                                   } }
+  ]
+}

salt/elasticsearch/files/ingest/zeek.dhcp

@@ -1,20 +1,21 @@
 {
-  "description" : "bro_dhcp",
+  "description" : "zeek.dhcp",
   "processors" : [
+    { "remove":         { "field": ["host"],     "ignore_failure": true                                                                  } }, 
     { "json":		{ "field": "message",			"target_field": "message2",		"ignore_failure": true	} },
-    { "rename": 	{ "field": "message2.uids", 	 	"target_field": "uid",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.mac", 		"target_field": "mac",			"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.assigned_ip",	"target_field": "assigned_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.lease_time", 	"target_field": "lease_time",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.trans_id", 	"target_field": "transaction_id",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.assigned_addr", 	"target_field": "assigned_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.client_addr", 	"target_field": "source_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.server_addr", 	"target_field": "destination_ip",	"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.requested_addr", 	"target_field": "requested_ip",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.domain", 		"target_field": "domain_name",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.host_name",	"target_field": "hostname",		"ignore_missing": true 	} },
-    { "rename": 	{ "field": "message2.duration",		"target_field": "duration",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.uids", 	 	"target_field": "log.id.uids",			"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.mac", 		"target_field": "host.mac",			"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.assigned_ip",	"target_field": "dhcp.assigned_ip",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.lease_time", 	"target_field": "dhcp.lease_time",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.trans_id", 	"target_field": "dhcp.transaction_id",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.assigned_addr", 	"target_field": "dhcp.assigned_ip",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.client_addr", 	"target_field": "client.address",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.server_addr", 	"target_field": "server.address",	"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.requested_addr", 	"target_field": "dhcp.requested_address",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.domain", 		"target_field": "host.domain",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.host_name",	"target_field": "host.hostname",		"ignore_missing": true 	} },
+    { "rename": 	{ "field": "message2.duration",		"target_field": "event.duration",		"ignore_missing": true 	} },
     { "rename": 	{ "field": "message2.msg_types", 	"target_field": "message_types",	"ignore_missing": true 	} },
-    { "pipeline":       { "name": "bro_common"                                                                                   } }
+    { "pipeline":       { "name": "zeek.common"                                                                                   } }
   ]
 }