8080e05444
on fresh install kafka nodes pillar may not have populated. Avoiding this by only generating kafka input pipeline when kafka nodes pillar is not empty
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-14 14:17:26 -04:00
f372b0907b
Use kafka:password for kafka certs
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-12 15:41:10 -04:00
e8106befe9
Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-12 12:05:16 -04:00
b1beb617b3
Logstash should be disabled when Kafka is enabled except when a minion override exists OR node is a standalone
...
- Standalone subscribes to Kafka topics via logstash for ingest
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-22 13:38:09 -04:00
2ad87bf1fe
merge 2.4/dev
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-08 16:30:45 -04:00
eca2a4a9c8
Logstash consumer threads should match topic partition count
...
- Default is set to 3. If there are too many consumer threads it may lead to idle logstash worker threads and could require decreasing this value to saturate workers
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-08 16:17:09 -04:00
dcc1f656ee
predownload logstash and elastic for new searchnode and heavynode
2024-05-07 10:13:51 -04:00
e960ae66a3
Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka
2024-05-02 15:12:27 -04:00
3efdb4e532
Reconfigure logstash Kafka input
...
- TODO: Configure what topics are pulled to searchnodes via the SOC UI
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 13:01:29 -04:00
84db82852c
annotation updates for custom settings
2024-04-30 15:14:56 -04:00
af53dcda1b
Remove references to kafkanode
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-11 15:32:00 -04:00
d3bd56b131
disable logstash and redis if kafka enabled
2024-04-10 14:13:27 -04:00
d67ebabc95
Remove logstash output to kafka pipeline. Add additional topics for searchnodes to ingest and add partition/offset info to event
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-08 16:38:03 -04:00
65274e89d7
Add client_id to logstash pipeline. To identify which searchnode is pulling messages
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-05 15:38:00 -04:00
721e04f793
initial logstash input from kafka over ssl
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-05 13:37:14 -04:00
82830c8173
Fix typos and fix error related to elasticsearch saltstate being called from logstash state. Logstash will be removed from kafkanodes in future
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-02 16:37:39 -04:00
446f1ffdf5
merge 2.4/dev
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-03-25 13:55:48 -04:00
1818e134ca
Change numbers for Logstash
2024-02-01 14:01:55 +00:00
136097f981
Custom Logstash pipeline annotations
2024-01-31 21:47:09 +00:00
4672a5b8eb
Custom pipeline configuration in UI
2024-01-31 20:18:17 +00:00
1853dc398b
Custom pipeline configuration
2024-01-31 20:17:33 +00:00
33a9ac5701
use logstash nodes for logstash extra_hosts
2023-12-15 15:42:49 -05:00
03b2a7d2de
change 9805 pipeline to send to self. fix extra_hosts for logstash
2023-12-14 10:01:03 -05:00
8cf29682bb
Update to merge in 2.4/dev
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2023-11-29 13:41:23 -05:00
86dc7cc804
Kafka init
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2023-11-29 13:34:25 -05:00
8c7767b381
Dont overwrite metadata
2023-11-03 08:41:33 -04:00
4dc64400c5
Support document_id
2023-11-01 13:36:32 +00:00
8a751e097d
cert path refactor
2023-08-23 14:32:05 -04:00
2472d6a727
Don't watch certs on search nodes
2023-08-03 18:52:29 -04:00
1c8a8c460c
Restart logstash when certs change
2023-08-02 17:53:29 -04:00
b6dd347eb8
Heavy Node add manager
2023-07-31 15:22:29 -04:00
78db64a419
Auto-managed Fleet Server URLs
2023-07-14 08:40:26 -04:00
c99e7da5a7
Remove Comments
2023-07-11 10:26:18 -04:00
31edf2e8ea
Tighten & Document Pipelines
2023-07-10 14:17:42 -04:00
7805ca8beb
Add Failover Support
2023-07-10 10:38:14 -04:00
8c16feb772
Rename Fleet pipelines
2023-07-09 12:22:55 -04:00
e8860a7d2c
Fix perms
2023-07-08 09:04:55 -04:00
beb26596fd
Merge remote-tracking branch 'origin/2.4/dev' into 2.4/fleetautogen
2023-07-07 19:12:47 -04:00
ff3bb11fbb
Elastic Fleet Certs Refactor
2023-07-07 16:44:16 -04:00
55bed0771b
remove so-logstash-get-unparsed, use so-redis-count instead
2023-07-07 09:52:21 -04:00
cb8faf7c5f
Fix the rest of the analyst entries
2023-06-26 16:14:04 -04:00
36272efda7
create ES_LOGSTASH_NODES which removes heavynodes
2023-06-22 09:46:42 -04:00
469390696e
2.4 receiver changes
2023-06-15 11:04:16 -04:00
2c4eccd7e0
2.4 heavynode changes
2023-06-14 10:40:05 -04:00
66dc6274e6
exclude elasticsearch.ca state from fleet and receiver nodes
2023-05-31 15:59:36 -04:00
5315c51197
Allow additional docker parameters
2023-05-18 16:52:38 -04:00
7ab31e36af
Merge branch '2.4/dev' of https://github.com/Security-Onion-Solutions/securityonion into airgaps
2023-05-18 15:19:15 -04:00
0fd9fb9294
Allow additional docker parameters
2023-05-18 15:19:09 -04:00
7d0251952c
Filter out uneeded Logstash metadata
2023-05-17 11:06:16 -04:00
24445cf36a
Rename Fleet pipelines
2023-05-16 16:43:21 -04:00
reyesj2
m0duspwnens
Jason Ertel
Wes
Josh Brower
Jason Ertel
Mike Reeves