Merge branch 'dev' of https://github.com/Security-Onion-Solutions/securityonion-saltstack into dev

README.md

@@ -1,37 +1,39 @@
-## Hybrid Hunter Alpha 1.1.4 - Feature Parity Release
+## Hybrid Hunter Beta 1.2.1 - Beta 1
 
 ### Changes:
 
-- Added new in-house auth method [Security Onion Auth](https://github.com/Security-Onion-Solutions/securityonion-auth).
-- Web user creation is done via the browser now instead of so-user-add.
-- New Logstash pipeline setup. Now uses multiple pipelines.
-- New Master + Search node type and well as a Heavy Node type in the install. 
-- Change all nodes to point to the docker registry on the Master. This cuts down on the calls to dockerhub.
-- Zeek 3.0.1
-- Elastic 6.8.6
-- New SO Start | Stop | Restart scripts for all components (eg. `so-playbook-restart`).
-- BPF support for Suricata (NIDS), Steno (PCAP) & Zeek ([Docs](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/BPF)).
-- Updated Domain Stats & Frequency Server containers to Python3 & created new Salt states for them.
-- Added so-status script which gives an easy to read look at container status.
-- Manage threshold.conf for Suricata using the thresholding pillar.
-- The ISO now includes all the docker containers for faster install speeds.
-- You now set the password for the onion account during the iso install. This account is temporary and will be removed after so-setup. 
-- Updated Helix parsers for better compatibility.
-- Updated telegraf docker to include curl and jq.
-- CVE-2020-0601 Zeek Detection Script. 
-- ISO Install now prompts you to create a password for the onion user during imaging. This account gets disabled during setup.
+- Full support for Ubuntu 18.04. 16.04 is no longer supported for Hybrid Hunter.
+- Introduction of the Security Onion Console. Once logged in you are directly taken to the SOC.
+- New authentication using Kratos.
+- During install you must specify how you would like to access the SOC ui. This is for strict cookie security.
+- Ability to list and delete web users from the SOC ui.
+- The soremote account is now used to add nodes to the grid vs using socore. 
+- Community ID support for Zeek, osquery, and Suricata. You can now tie host events to connection logs!
+- Elastic 7.6.1 with ECS support.
+- New set of Kibana dashboards that align with ECS.
+- Eval mode no longer uses Logstash for parsing (Filebeat -> ES Ingest)
+- Ingest node parsing for osquery-shipped logs (osquery, WEL, Sysmon).
+- Fleet standalone mode with improved Web UI & API access control.
+- Improved Fleet integration support.
+- Playbook now has full Windows Sigma community ruleset builtin.
+- Automatic Sigma community rule updates.
+- Playbook stability enhancements.
+- Zeek health check. Zeek will now auto restart if a worker crashes.
+- zeekctl is now managed by salt.
+- Grafana dashboard improvements and cleanup.
+- Moved logstash configs to pillars.
+- Salt logs moved to /opt/so/log/salt.
+- Strelka integrated for file-oriented detection/analysis at scale
 
-## Version 1.1.4 ISO Download
+### Known issues:
 
-[HH1.1.4-46.ISO](https://download.securityonion.net/file/Hybrid-Hunter/HH-1.1.4-46.iso)  
-
-MD5: ACF6B4586E8EE7D1938FB2C028DFC987  
-SHA1: C29B4F3748604196357EC7262BF071177E696D86  
-SHA256: 4D977B650196441294D53372F248B50C23E933B8FBEC5CC5BAB569DFEF31E7E8  
+- Updating users via the SOC ui is known to fail. To change a user, delete the user and re-add them. 
+- Due to the move to ECS, the current Playbook plays may not alert correctly at this time.
+- The osquery MacOS package does not install correctly.
 
 ### Warnings and Disclaimers
 
-- This ALPHA release is BLEEDING EDGE and TOTALLY UNSUPPORTED!  
+- This BETA release is BLEEDING EDGE and TOTALLY UNSUPPORTED!  
 - If this breaks your system, you get to keep both pieces!  
 - This script is a work in progress and is in constant flux.  
 - This script is intended to build a quick prototype proof of concept so you can see what our new platform might look like.  This configuration will change drastically over time leading up to the final release.  
@@ -44,33 +46,36 @@ SHA256: 4D977B650196441294D53372F248B50C23E933B8FBEC5CC5BAB569DFEF31E7E8
 
 Evaluation Mode:
 
-- ISO or a Single VM running Ubuntu 16.04 or CentOS 7
+- ISO or a Single VM running Ubuntu 18.04 or CentOS 7
 - Minimum 12GB of RAM
 - Minimum 4 CPU cores
 - Minimum 2 NICs
 
 Distributed:
 
-- 3 VMs running the ISO or Ubuntu 16.04 or CentOS 7 (You can mix and match)
+- 3 VMs running the ISO or Ubuntu 18.04 or CentOS 7 (You can mix and match)
 - Minimum 8GB of RAM per VM
 - Minimum 4 CPU cores per VM
 - Minimum 2 NICs for forward nodes
 
-### Prerequisites for Network Based Install
+### Installation
 
-Install git if using a Centos 7 Minimal install:
+For most users, we recommend installing using [our ISO image](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/ISO).
+
+If instead you would like to try a manual installation (not using our ISO), you can build from CentOS 7 or Ubuntu 18.04.
+
+If using CentOS 7 Minimal, you will need to install git:
 
 ```sudo yum -y install git```
 
-### Installation
-
-Once you resolve those requirements or are using Ubuntu 16.04 do the following:
+Once you have git, then do the following:
 
 ```
 git clone https://github.com/Security-Onion-Solutions/securityonion-saltstack
 cd securityonion-saltstack
 sudo bash so-setup-network
 ```
+
 Follow the prompts and reboot if asked to do so.
 
 Then proceed to the [Hybrid Hunter Quick Start Guide](https://github.com/Security-Onion-Solutions/securityonion-saltstack/wiki/Hybrid-Hunter-Quick-Start-Guide).

pillar/top.sls

@@ -34,8 +34,6 @@ base:
     - data.*
     - brologs
     - secrets
-    - logstash
-    - logstash.eval
     - healthcheck.eval
     - minions.{{ grains.id }}
 

salt/registry/etc/config.yml

@@ -19,5 +19,4 @@ health:
     enabled: true
     interval: 10s
     threshold: 3
-proxy:
-  remoteurl: https://registry-1.docker.io
+

setup/so-functions

@@ -603,9 +603,9 @@ docker_seed_registry() {
     "so-soctopus:$VERSION" \
     "so-steno:$VERSION" \
     "so-strelka-frontend:$VERSION" \
-	"so-strelka-manager:$VERSION" \
-	"so-strelka-backend:$VERSION" \
-	"so-strelka-filestream:$VERSION" \
+    "so-strelka-manager:$VERSION" \
+    "so-strelka-backend:$VERSION" \
+    "so-strelka-filestream:$VERSION" \
     "so-suricata:$VERSION" \
     "so-telegraf:$VERSION" \
     "so-thehive:$VERSION" \
@@ -645,6 +645,8 @@ docker_seed_registry() {
     done
   else
     # We already have the goods son
+    cd /nsm/docker-registry/docker
+    tar xvf so-dockers-$VERSION.tar
     rm /nsm/docker-registry/docker/so-dockers-$VERSION.tar
   fi
 
@@ -1323,6 +1325,7 @@ EOF
         echo "deb http://repo.saltstack.com/apt/ubuntu/$UVER/amd64/2019.2 $OSVER main" > /etc/apt/sources.list.d/saltstack.list
       fi
 	  echo "deb https://packages.wazuh.com/3.x/apt/ stable main" | tee /etc/apt/sources.list.d/wazuh.list
+    
       # Initialize the new repos
       apt-get update >> $SETUPLOG 2>&1
       if [ $OSVER != "xenial" ]; then