reyesj2
dd17ee7665
fix defining custom logstash pipelines when kafka is enabled
2025-02-06 22:19:24 -06:00
defensivedepth
4c5099d429
Initial support for local lookup
2024-10-29 10:27:54 -04:00
m0duspwnens
5fb660bc9a
remove kernel bool option, just use list
2024-10-17 09:29:03 -04:00
Jason Ertel
217bb388a0
Clarify enabled settings
2024-09-16 10:05:17 -04:00
m0duspwnens
8702d95434
only elasticsearch image uses es version
2024-09-09 16:38:38 -04:00
m0duspwnens
5a1d61a042
ref es version
2024-09-05 08:45:44 -04:00
m0duspwnens
cd9c9a25d3
reference elastic versions from defaults
2024-08-21 11:25:56 -04:00
reyesj2
7ea8d5efd0
Remove redis input pipeline from searchnodes when global pipeline is Kafka
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-07-12 14:44:10 -04:00
reyesj2
4182ff66a0
rearrange kafka pillar, declutters SOC ui
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-07-11 16:37:16 -04:00
reyesj2
d5faf535c3
Only interact with logstash configuration when Kafka pipeline is enabled otherwise leave it default
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-07-10 11:36:44 -04:00
reyesj2
8e1edd1d91
split Kafka ssl from ssl/init. Certs won't be generated until Kafka is enabled. Also runs some clean up for old Kafka certs
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-07-10 11:32:43 -04:00
reyesj2
d791b23838
Generate new Kafka truststore
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-07-10 11:29:09 -04:00
m0duspwnens
50f0c43212
merge dev
2024-06-26 12:33:32 -04:00
reyesj2
c332cd777c
remove import/heavynode artifact caused by kafka cert not existing but being bound in docker. (empty dir created)
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-24 08:50:37 -04:00
m0duspwnens
469ca44016
fix maps
2024-06-20 16:53:12 -04:00
m0duspwnens
81fcd68e9b
create and use redis:nodes and elasticsearch:nodes pillars
2024-06-20 16:42:11 -04:00
m0duspwnens
55f8303dc2
remove manager and search pipelines from heavynode
2024-06-17 10:06:43 -04:00
reyesj2
0b1175b46c
kafka logstash input plugin handle empty brokers list
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-14 23:03:36 -04:00
reyesj2
8080e05444
on fresh install kafka nodes pillar may not have populated. Avoiding this by only generating kafka input pipeline when kafka nodes pillar is not empty
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-14 14:17:26 -04:00
reyesj2
f372b0907b
Use kafka:password for kafka certs
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-12 15:41:10 -04:00
reyesj2
e8106befe9
Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-06-12 12:05:16 -04:00
reyesj2
b1beb617b3
Logstash should be disabled when Kafka is enabled except when a minion override exists OR node is a standalone
...
- Standalone subscribes to Kafka topics via logstash for ingest
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-22 13:38:09 -04:00
reyesj2
2ad87bf1fe
merge 2.4/dev
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-08 16:30:45 -04:00
reyesj2
eca2a4a9c8
Logstash consumer threads should match topic partition count
...
- Default is set to 3. If there are too many consumer threads it may lead to idle logstash worker threads and could require decreasing this value to saturate workers
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-08 16:17:09 -04:00
m0duspwnens
dcc1f656ee
predownload logstash and elastic for new searchnode and heavynode
2024-05-07 10:13:51 -04:00
reyesj2
e960ae66a3
Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka
2024-05-02 15:12:27 -04:00
reyesj2
3efdb4e532
Reconfigure logstash Kafka input
...
- TODO: Configure what topics are pulled to searchnodes via the SOC UI
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 13:01:29 -04:00
Jason Ertel
84db82852c
annotation updates for custom settings
2024-04-30 15:14:56 -04:00
reyesj2
af53dcda1b
Remove references to kafkanode
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-11 15:32:00 -04:00
m0duspwnens
d3bd56b131
disable logstash and redis if kafka enabled
2024-04-10 14:13:27 -04:00
reyesj2
d67ebabc95
Remove logstash output to kafka pipeline. Add additional topics for searchnodes to ingest and add partition/offset info to event
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-08 16:38:03 -04:00
reyesj2
65274e89d7
Add client_id to logstash pipeline. To identify which searchnode is pulling messages
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-05 15:38:00 -04:00
reyesj2
721e04f793
initial logstash input from kafka over ssl
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-05 13:37:14 -04:00
reyesj2
82830c8173
Fix typos and fix error related to elasticsearch saltstate being called from logstash state. Logstash will be removed from kafkanodes in future
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-02 16:37:39 -04:00
reyesj2
446f1ffdf5
merge 2.4/dev
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-03-25 13:55:48 -04:00
Wes
1818e134ca
Change numbers for Logstash
2024-02-01 14:01:55 +00:00
Wes
136097f981
Custom Logstash pipeline annotations
2024-01-31 21:47:09 +00:00
Wes
4672a5b8eb
Custom pipeline configuration in UI
2024-01-31 20:18:17 +00:00
Wes
1853dc398b
Custom pipeline configuration
2024-01-31 20:17:33 +00:00
m0duspwnens
33a9ac5701
use logstash nodes for logstash extra_hosts
2023-12-15 15:42:49 -05:00
m0duspwnens
03b2a7d2de
change 9805 pipeline to send to self. fix extra_hosts for logstash
2023-12-14 10:01:03 -05:00
reyesj2
8cf29682bb
Update to merge in 2.4/dev
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2023-11-29 13:41:23 -05:00
reyesj2
86dc7cc804
Kafka init
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2023-11-29 13:34:25 -05:00
Josh Brower
8c7767b381
Dont overwrite metadata
2023-11-03 08:41:33 -04:00
Wes
4dc64400c5
Support document_id
2023-11-01 13:36:32 +00:00
Jason Ertel
8a751e097d
cert path refactor
2023-08-23 14:32:05 -04:00
Josh Brower
2472d6a727
Don't watch certs on search nodes
2023-08-03 18:52:29 -04:00
Josh Brower
1c8a8c460c
Restart logstash when certs change
2023-08-02 17:53:29 -04:00
Josh Brower
b6dd347eb8
Heavy Node add manager
2023-07-31 15:22:29 -04:00
Josh Brower
78db64a419
Auto-managed Fleet Server URLs
2023-07-14 08:40:26 -04:00
