CSEC_PUBLIC/securityonion
1
0
Fork 0
mirror of https://github.com/Security-Onion-Solutions/securityonion.git synced 2025-12-06 09:12:45 +01:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'remotes/origin/dev' into issue/140

Browse Source
This commit is contained in:
m0duspwnens
2020-05-15 15:21:03 -04:00
parent e7baca16ef e02bf2ebb5
commit 4e63477b98
14 changed files with 127 additions and 90 deletions
Download Patch File Download Diff File Expand all files Collapse all files

50
README.md
View File

@@ -1,32 +1,34 @@
## Hybrid Hunter Beta 1.2.1 - Beta 1
## Hybrid Hunter Beta 1.3.0 - Beta 2
### Changes:
- Full support for Ubuntu 18.04. 16.04 is no longer supported for Hybrid Hunter.
- Introduction of the Security Onion Console. Once logged in you are directly taken to the SOC.
- New authentication using Kratos.
- During install you must specify how you would like to access the SOC ui. This is for strict cookie security.
- Ability to list and delete web users from the SOC ui.
- The soremote account is now used to add nodes to the grid vs using socore.
- Community ID support for Zeek, osquery, and Suricata. You can now tie host events to connection logs!
- Elastic 7.6.1 with ECS support.
- New set of Kibana dashboards that align with ECS.
- Eval mode no longer uses Logstash for parsing (Filebeat -> ES Ingest)
- Ingest node parsing for osquery-shipped logs (osquery, WEL, Sysmon).
- Fleet standalone mode with improved Web UI & API access control.
- Improved Fleet integration support.
- Playbook now has full Windows Sigma community ruleset builtin.
- Automatic Sigma community rule updates.
- Playbook stability enhancements.
- Zeek health check. Zeek will now auto restart if a worker crashes.
- zeekctl is now managed by salt.
- Grafana dashboard improvements and cleanup.
- Moved logstash configs to pillars.
- Salt logs moved to /opt/so/log/salt.
- Strelka integrated for file-oriented detection/analysis at scale
- New Feature: Codename: "Onion Hunt". Select Hunt from the menu and start hunting down your adversaries!
- Improved ECS support.
- Complete refactor of the setup to make it easier to follow.
- Improved setup script logging to better assist on any issues.
- Setup now checks for minimal requirements during install.
- Updated Cyberchef to version 9.20.3.
- Updated Elastalert to version 0.2.4 and switched to alpine to reduce container size.
- Updated Redis to 5.0.9 and switched to alpine to reduce container size.
- Updated Salt to 2019.2.5
- Updated Grafana to 6.7.3.
- Zeek 3.0.6
- Suricata 4.1.8
- Fixes so-status to now display correct containers and status.
- local.zeek is now controlled by a pillar instead of modifying the file directly.
- Renamed so-core to so-nginx and switched to alpine to reduce container size.
- Playbook now uses MySQL instead of SQLite.
- Sigma rules have all been updated.
- Kibana dashboard improvements for ECS.
- Fixed an issue where geoip was not properly parsed.
- ATT&CK Navigator is now it's own state.
- Standlone mode is now supported.
- Mastersearch previously used the same Grafana dashboard as a Search node. It now has its own dashboard that incorporates panels from the Master node and Search node dashboards.
### Known issues:
### Known Issues:
- The Hunt feature is currently considered "Preview" and although very useful in its current state, not everything works. We wanted to get this out as soon as possible to get the feedback from you! Let us know what you want to see! Let us know what you think we should call it!
- You cannot pivot to PCAP from Suricata alerts in Kibana or Hunt.
- Updating users via the SOC ui is known to fail. To change a user, delete the user and re-add them.
- Due to the move to ECS, the current Playbook plays may not alert correctly at this time.
- The osquery MacOS package does not install correctly.

4
salt/elasticsearch/files/ingest/common
View File

@@ -41,8 +41,8 @@
{ "rename": { "field": "message2.community_id", "target_field": "network.community_id", "ignore_missing": true } },
{
"remove": {
"field": [ "index_name_prefix", "message2"],
"ignore_failure": false
"field": [ "index_name_prefix", "message2", "type" ],
"ignore_failure": true
}
}
]

8
salt/elasticsearch/files/ingest/osquery.query_result
View File

@@ -24,8 +24,14 @@
{ "rename": { "field": "message3.columns.pid", "target_field": "process.pid", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.parent", "target_field": "process.ppid", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.cwd", "target_field": "process.working_directory", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.community_id", "target_field": "network.community_id", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.local_address", "target_field": "local.ip", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.local_port", "target_field": "local.port", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.remote_address", "target_field": "remote.ip", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.remote_port", "target_field": "remote.port", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.process_name", "target_field": "process.name", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.eventid", "target_field": "event.code", "ignore_missing": true } },
{ "set": { "if": "ctx.message3.columns.data != null", "field": "dataset", "value": "wel-{{message3.columns.source}}", "override": true } },
{ "set": { "if": "ctx.message3.columns.?data != null", "field": "dataset", "value": "wel-{{message3.columns.source}}", "override": true } },
{ "rename": { "field": "message3.columns.winlog.EventData.SubjectUserName", "target_field": "user.name", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.winlog.EventData.destinationHostname", "target_field": "destination.hostname", "ignore_missing": true } },
{ "rename": { "field": "message3.columns.winlog.EventData.destinationIp", "target_field": "destination.ip", "ignore_missing": true } },

4
salt/elasticsearch/files/ingest/zeek.common
View File

@@ -12,9 +12,9 @@
{ "rename": { "field": "message2.id.resp_h", "target_field": "destination.ip", "ignore_missing": true } },
{ "rename": { "field": "message2.id.resp_p", "target_field": "destination.port", "ignore_missing": true } },
{ "set": { "field": "client.ip", "value": "{{source.ip}}" } },
{ "set": { "if": "ctx.source.port != null", "field": "client.port", "value": "{{source.port}}" } },
{ "set": { "if": "ctx.source?.port != null", "field": "client.port", "value": "{{source.port}}" } },
{ "set": { "field": "server.ip", "value": "{{destination.ip}}" } },
{ "set": { "if": "ctx.destination.port != null", "field": "server.port", "value": "{{destination.port}}" } },
{ "set": { "if": "ctx.destination?.port != null", "field": "server.port", "value": "{{destination.port}}" } },
{ "set": { "field": "observer.name", "value": "{{agent.name}}" } },
{ "date": { "field": "message2.ts", "target_field": "@timestamp", "formats": ["ISO8601", "UNIX"], "ignore_failure": true } },
{ "remove": { "field": ["agent"], "ignore_failure": true } },

14
salt/elasticsearch/files/ingest/zeek.conn
View File

@@ -21,6 +21,20 @@
{ "rename": { "field": "message2.orig_cc", "target_field": "client.country_code","ignore_missing": true } },
{ "rename": { "field": "message2.resp_cc", "target_field": "server.country_code", "ignore_missing": true } },
{ "rename": { "field": "message2.sensorname", "target_field": "observer.name", "ignore_missing": true } },
{ "script": { "lang": "painless", "source": "ctx.network.bytes = (ctx.client.bytes + ctx.server.bytes)", "ignore_failure": true } },
{ "set": { "if": "ctx.connection.state == 'S0'", "field": "connection.state_description", "value": "Connection attempt seen, no reply" } },
{ "set": { "if": "ctx.connection.state == 'S1'", "field": "connection.state_description", "value": "Connection established, not terminated" } },
{ "set": { "if": "ctx.connection.state == 'S2'", "field": "connection.state_description", "value": "Connection established and close attempt by originator seen (but no reply from responder)" } },
{ "set": { "if": "ctx.connection.state == 'S3'", "field": "connection.state_description", "value": "Connection established and close attempt by responder seen (but no reply from originator)" } },
{ "set": { "if": "ctx.connection.state == 'SF'", "field": "connection.state_description", "value": "Normal SYN/FIN completion" } },
{ "set": { "if": "ctx.connection.state == 'REJ'", "field": "connection.state_description", "value": "Connection attempt rejected" } },
{ "set": { "if": "ctx.connection.state == 'RSTO'", "field": "connection.state_description", "value": "Connection established, originator aborted (sent a RST)" } },
{ "set": { "if": "ctx.connection.state == 'RSTR'", "field": "connection.state_description", "value": "Established, responder aborted" } },
{ "set": { "if": "ctx.connection.state == 'RSTOS0'","field": "connection.state_description", "value": "Originator sent a SYN followed by a RST, we never saw a SYN-ACK from the responder" } },
{ "set": { "if": "ctx.connection.state == 'RSTRH'", "field": "connection.state_description", "value": "Responder sent a SYN ACK followed by a RST, we never saw a SYN from the (purported) originator" } },
{ "set": { "if": "ctx.connection.state == 'SH'", "field": "connection.state_description", "value": "Originator sent a SYN followed by a FIN, we never saw a SYN ACK from the responder (hence the connection was 'half' open)" } },
{ "set": { "if": "ctx.connection.state == 'SHR'", "field": "connection.state_description", "value": "Responder sent a SYN ACK followed by a FIN, we never saw a SYN from the originator" } },
{ "set": { "if": "ctx.connection.state == 'OTH'", "field": "connection.state_description", "value": "No SYN seen, just midstream traffic (a 'partial connection' that was not later closed)" } },
{ "pipeline": { "name": "zeek.common" } }
]
}

6
salt/elasticsearch/files/ingest/zeek.dce_rpc
View File

@@ -4,9 +4,9 @@
{ "remove": { "field": ["host"], "ignore_failure": true } },
{ "json": { "field": "message", "target_field": "message2", "ignore_failure": true } },
{ "rename": { "field": "message2.rtt", "target_field": "event.duration", "ignore_missing": true } },
{ "rename": { "field": "message2.named_pipe", "target_field": "named_pipe", "ignore_missing": true } },
{ "rename": { "field": "message2.endpoint", "target_field": "endpoint", "ignore_missing": true } },
{ "rename": { "field": "message2.operation", "target_field": "operation", "ignore_missing": true } },
{ "rename": { "field": "message2.named_pipe", "target_field": "dce_rpc.named_pipe", "ignore_missing": true } },
{ "rename": { "field": "message2.endpoint", "target_field": "dce_rpc.endpoint", "ignore_missing": true } },
{ "rename": { "field": "message2.operation", "target_field": "dce_rpc.operation", "ignore_missing": true } },
{ "pipeline": { "name": "zeek.common" } }
]
}

11
salt/firewall/init.sls
View File

@@ -365,6 +365,17 @@ enable_minion_osquery_8080_{{ip}}:
- position: 1
- save: True
enable_minion_osquery_8090_{{ip}}:
iptables.insert:
- table: filter
- chain: DOCKER-USER
- jump: ACCEPT
- proto: tcp
- source: {{ ip }}
- dport: 8090
- position: 1
- save: True
enable_minion_wazuh_55000_{{ip}}:
iptables.insert:
- table: filter

4
salt/fleet/files/scripts/so-fleet-setup
View File

@@ -31,7 +31,7 @@ docker exec so-fleet fleetctl apply -f /packs/hh/osquery.conf
# Enable Fleet
echo "Enabling Fleet..."
salt-call state.apply fleet.event_enable-fleet queue=True >> /root/fleet-setup.log
salt-call state.apply common queue=True >> /root/fleet-setup.log
salt-call state.apply nginx queue=True >> /root/fleet-setup.log
# Generate osquery install packages
echo "Generating osquery install packages - this will take some time..."
@@ -42,7 +42,7 @@ echo "Installing launcher via salt..."
salt-call state.apply fleet.install_package queue=True >> /root/fleet-setup.log
salt-call state.apply filebeat queue=True >> /root/fleet-setup.log
docker stop so-nginx
salt-call state.apply common queue=True >> /root/fleet-setup.log
salt-call state.apply nginx queue=True >> /root/fleet-setup.log
echo "Fleet Setup Complete - Login here: https://{{ MAIN_HOSTNAME }}"
echo "Your username is $2 and your password is $initpw"

48
salt/soc/files/soc/changes.json
View File

@@ -1,28 +1,30 @@
{
"title": "Introducing Hybrid Hunter 1.2.1 Beta 1",
"title": "Introducing Hybrid Hunter 1.3.0 Beta 2",
"changes": [
{ "summary": "Full support for Ubuntu 18.04. 16.04 is no longer supported for Hybrid Hunter." },
{ "summary": "Introduction of the Security Onion Console. Once logged in you are directly taken to the SOC." },
{ "summary": "New authentication using Kratos." },
{ "summary": "During install you must specify how you would like to access the SOC ui. This is for strict cookie security." },
{ "summary": "Ability to list and delete web users from the SOC ui." },
{ "summary": "The soremote account is now used to add nodes to the grid vs using socore." },
{ "summary": "Community ID support for Zeek, osquery, and Suricata. You can now tie host events to connection logs!" },
{ "summary": "Elastic 7.6.1 with ECS support." },
{ "summary": "New set of Kibana dashboards that align with ECS." },
{ "summary": "Eval mode no longer uses Logstash for parsing (Filebeat -> ES Ingest)" },
{ "summary": "Ingest node parsing for osquery-shipped logs (osquery, WEL, Sysmon)." },
{ "summary": "Fleet standalone mode with improved Web UI & API access control." },
{ "summary": "Improved Fleet integration support." },
{ "summary": "Playbook now has full Windows Sigma community ruleset builtin." },
{ "summary": "Automatic Sigma community rule updates." },
{ "summary": "Playbook stability enhancements." },
{ "summary": "Zeek health check. Zeek will now auto restart if a worker crashes." },
{ "summary": "zeekctl is now managed by salt." },
{ "summary": "Grafana dashboard improvements and cleanup." },
{ "summary": "Moved logstash configs to pillars." },
{ "summary": "Salt logs moved to /opt/so/log/salt." },
{ "summary": "Strelka integrated for file-oriented detection/analysis at scale" },
{ "summary": "New Feature: Codename: \"Onion Hunt\". Select Hunt from the menu and start hunting down your adversaries!" },
{ "summary": "Improved ECS support." },
{ "summary": "Complete refactor of the setup to make it easier to follow." },
{ "summary": "Improved setup script logging to better assist on any issues." },
{ "summary": "Setup now checks for minimal requirements during install." },
{ "summary": "Updated Cyberchef to version 9.20.3." },
{ "summary": "Updated Elastalert to version 0.2.4 and switched to alpine to reduce container size." },
{ "summary": "Updated Redis to 5.0.9 and switched to alpine to reduce container size." },
{ "summary": "Updated Salt to 2019.2.5." },
{ "summary": "Updated Grafana to 6.7.3." },
{ "summary": "Zeek 3.0.6." },
{ "summary": "Suricata 4.1.8." },
{ "summary": "Fixes so-status to now display correct containers and status." },
{ "summary": "local.zeek is now controlled by a pillar instead of modifying the file directly." },
{ "summary": "Renamed so-core to so-nginx and switched to alpine to reduce container size." },
{ "summary": "Playbook now uses MySQL instead of SQLite." },
{ "summary": "Sigma rules have all been updated." },
{ "summary": "Kibana dashboard improvements for ECS." },
{ "summary": "Fixed an issue where geoip was not properly parsed." },
{ "summary": "ATT&CK Navigator is now it's own state." },
{ "summary": "Standlone mode is now supported." },
{ "summary": "Mastersearch previously used the same Grafana dashboard as a Search node. It now has its own dashboard that incorporates panels from the Master node and Search node dashboards." },
{ "summary": "KNOWN ISSUE: The Hunt feature is currently considered \"Preview\" and although very useful in its current state, not everything works. We wanted to get this out as soon as possible to get the feedback from you! Let us know what you want to see! Let us know what you think we should call it!" },
{ "summary": "KNOWN ISSUE: You cannot pivot to PCAP from Suricata alerts in Kibana or Hunt." },
{ "summary": "KNOWN ISSUE: Updating users via the SOC ui is known to fail. To change a user, delete the user and re-add them." },
{ "summary": "KNOWN ISSUE: Due to the move to ECS, the current Playbook plays may not alert correctly at this time." },
{ "summary": "KNOWN ISSUE: The osquery MacOS package does not install correctly." }

2
salt/soc/files/soc/soc.json
View File

@@ -88,7 +88,7 @@
{ "name": "Alerts", "description": "Show all alerts grouped by alert source", "query": "event.dataset: alert | groupby event.module"},
{ "name": "NIDS Alerts", "description": "Show all NIDS alerts grouped by alert name", "query": "event.category: network AND event.dataset: alert | groupby rule.name"},
{ "name": "Wazuh/OSSEC Alerts", "description": "Show all Wazuh alerts grouped by category", "query": "event.module:ossec AND event.dataset:alert | groupby rule.category"},
{ "name": "Wazuh/OSSEC Commands", "description": "Show all Wazuh alerts grouped by command line", "query": "eevent.module:ossec AND event.dataset:alert | groupby process.command_line"},
{ "name": "Wazuh/OSSEC Commands", "description": "Show all Wazuh alerts grouped by command line", "query": "event.module:ossec AND event.dataset:alert | groupby process.command_line"},
{ "name": "Wazuh/OSSEC Processes", "description": "Show all Wazuh alerts grouped by process name", "query": "event.module:ossec AND event.dataset:alert | groupby process.name"},
{ "name": "Wazuh/OSSEC Users", "description": "Show all Wazuh alerts grouped by username", "query": "event.module:ossec AND event.dataset:alert | groupby user.name"},
{ "name": "Sysmon Events", "description": "Show all Sysmon logs grouped by event_id", "query": "event_type:sysmon | groupby event_id"},

6
salt/utility/bin/crossthestreams
View File

@@ -38,9 +38,3 @@ echo "Applying cross cluster search config..."
curl -XPUT http://{{ ES }}:9200/_cluster/settings -H'Content-Type: application/json' -d '{"persistent": {"search": {"remote": {"{{ SN }}": {"skip_unavailable": "true", "seeds": ["{{ SNDATA.ip }}:9300"]}}}}}'
{%- endfor %}
{%- endif %}
{%- if salt['pillar.get']('mastersearchtab', {}) %}
{%- for SN, SNDATA in salt['pillar.get']('mastersearchtab', {}).items() %}
curl -XPUT http://{{ ES }}:9200/_cluster/settings -H'Content-Type: application/json' -d '{"persistent": {"search": {"remote": {"{{ SN }}": {"skip_unavailable": "true", "seeds": ["{{ SNDATA.ip }}:9300"]}}}}}'
{%- endfor %}
{%- endif %}

3
setup/so-functions
View File

@@ -686,8 +686,7 @@ docker_seed_registry() {
} >> "$setup_log" 2>&1
done
else
cd /nsm/docker-registry/docker
tar xvf /nsm/docker-registry/docker/registry.tar >> "$setup_log" 2>&1
tar xvf /nsm/docker-registry/docker/registry.tar -C /nsm/docker-registry/docker >> "$setup_log" 2>&1
rm /nsm/docker-registry/docker/registry.tar >> "$setup_log" 2>&1
fi

23
setup/so-setup
View File

@@ -116,13 +116,7 @@ case "$setup_type" in
whiptail_management_interface_dns_search
fi
# Init networking so rest of install works
set_hostname_iso
set_management_interface
collect_adminuser_inputs
add_admin_user
disable_onion_user
;;
'network')
whiptail_network_notice
@@ -247,6 +241,15 @@ fi
whiptail_make_changes
if [[ "$setup_type" == 'iso' ]]; then
# Init networking so rest of install works
set_hostname_iso
set_management_interface
add_admin_user
disable_onion_user
fi
set_hostname 2>> "$setup_log"
set_version 2>> "$setup_log"
clear_master 2>> "$setup_log"
@@ -317,7 +320,6 @@ export percentage=0
fi
set_progress_str 16 'Running first Salt checkin'
salt_firstcheckin 2>> "$setup_log"
@@ -355,7 +357,12 @@ export percentage=0
set_progress_str 25 'Configuring firewall'
set_initial_firewall_policy 2>> "$setup_log"
if [[ "$setup_type" == 'iso' ]]; then
set_progress_str 26 'Copying containers from iso'
else
set_progress_str 26 'Downloading containers from the internet'
fi
salt-call state.apply -l info registry >> "$setup_log" 2>&1
docker_seed_registry 2>> "$setup_log" # ~ 60% when finished
@@ -461,8 +468,10 @@ export percentage=0
set_progress_str 86 'Updating packages'
update_packages 2>> "$setup_log"
if [[ $is_master ]]; then
set_progress_str 87 'Adding user to SOC'
add_web_user 2>> "$setup_log"
fi
set_progress_str 90 'Enabling checkin at boot'
checkin_at_boot 2>> "$setup_log"

18
setup/so-whiptail
View File

@@ -429,7 +429,7 @@ whiptail_management_interface_dns() {
[ -n "$TESTING" ] && return
MDNS=$(whiptail --title "Security Onion Setup" --inputbox \
"Enter your DNS server using space between multiple" 10 60 8.8.8.8 8.8.4.4 3>&1 1>&2 2>&3)
"Enter your DNS servers separated by a space" 10 60 8.8.8.8 8.8.4.4 3>&1 1>&2 2>&3)
}
@@ -958,7 +958,7 @@ whiptail_setup_complete() {
[ -n "$TESTING" ] && return
whiptail --title "Security Onion Setup" --msgbox "Finished installing this as an $install_type. Press Enter to reboot." 8 75
whiptail --title "Security Onion Setup" --msgbox "Finished $install_type install. Press ENTER to reboot." 8 75
install_cleanup >> $setup_log 2>&1
}
@@ -967,7 +967,7 @@ whiptail_setup_failed() {
[ -n "$TESTING" ] && return
whiptail --title "Security Onion Setup" --msgbox "Install had a problem. Please see $setup_log for details. Press Enter to reboot." 8 75
whiptail --title "Security Onion Setup" --msgbox "Install had a problem. Please see $setup_log for details. Press ENTER to reboot." 8 75
install_cleanup >> $setup_log 2>&1
}
@@ -1012,9 +1012,9 @@ whiptail_master_updates() {
local update_string
update_string=$(whiptail --title "Security Onion Setup" --radiolist \
"How would you like to download updates for your grid?:" 20 75 4 \
"MASTER" "Master node is proxy for OS/Docker updates." ON \
"OPEN" "Each node connect to the Internet for updates" OFF 3>&1 1>&2 2>&3 )
"How would you like to download OS package updates for your grid?:" 20 75 4 \
"MASTER" "Master node is proxy for updates." ON \
"OPEN" "Each node connects to the Internet for updates" OFF 3>&1 1>&2 2>&3 )
local exitstatus=$?
whiptail_check_exitstatus $exitstatus
@@ -1035,9 +1035,9 @@ whiptail_node_updates() {
[ -n "$TESTING" ] && return
NODEUPDATES=$(whiptail --title "Security Onion Setup" --radiolist \
"How would you like to download updates for this node?:" 20 75 4 \
"MASTER" "Download OS/Docker updates from the Master." ON \
"OPEN" "Download updates directly from the Internet" OFF 3>&1 1>&2 2>&3 )
"How would you like to download OS package updates for your grid?:" 20 75 4 \
"MASTER" "Master node is proxy for updates." ON \
"OPEN" "Each node connects to the Internet for updates" OFF 3>&1 1>&2 2>&3 )
local exitstatus=$?
whiptail_check_exitstatus $exitstatus