Commit Graph

169 Commits

Author SHA1 Message Date
reyesj2
8080e05444 on fresh install kafka nodes pillar may not have populated. Avoiding this by only generating kafka input pipeline when kafka nodes pillar is not empty
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-06-14 14:17:26 -04:00
reyesj2
f372b0907b Use kafka:password for kafka certs
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-06-12 15:41:10 -04:00
reyesj2
e8106befe9 Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-06-12 12:05:16 -04:00
reyesj2
eca2a4a9c8 Logstash consumer threads should match topic partition count
- Default is set to 3. If there are too many consumer threads it may lead to idle logstash worker threads and could require decreasing this value to saturate workers

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-05-08 16:17:09 -04:00
reyesj2
3efdb4e532 Reconfigure logstash Kafka input
- TODO: Configure what topics are pulled to searchnodes via the SOC UI

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-05-01 13:01:29 -04:00
reyesj2
af53dcda1b Remove references to kafkanode
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-11 15:32:00 -04:00
reyesj2
d67ebabc95 Remove logstash output to kafka pipeline. Add additional topics for searchnodes to ingest and add partition/offset info to event
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-08 16:38:03 -04:00
reyesj2
65274e89d7 Add client_id to logstash pipeline. To identify which searchnode is pulling messages
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-05 15:38:00 -04:00
reyesj2
721e04f793 initial logstash input from kafka over ssl
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-05 13:37:14 -04:00
reyesj2
446f1ffdf5 merge 2.4/dev
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-03-25 13:55:48 -04:00
m0duspwnens
03b2a7d2de change 9805 pipeline to send to self. fix extra_hosts for logstash 2023-12-14 10:01:03 -05:00
reyesj2
8cf29682bb Update to merge in 2.4/dev
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2023-11-29 13:41:23 -05:00
reyesj2
86dc7cc804 Kafka init
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2023-11-29 13:34:25 -05:00
Josh Brower
8c7767b381 Dont overwrite metadata 2023-11-03 08:41:33 -04:00
Wes
4dc64400c5 Support document_id 2023-11-01 13:36:32 +00:00
Josh Brower
78db64a419 Auto-managed Fleet Server URLs 2023-07-14 08:40:26 -04:00
Josh Brower
31edf2e8ea Tighten & Document Pipelines 2023-07-10 14:17:42 -04:00
Josh Brower
7805ca8beb Add Failover Support 2023-07-10 10:38:14 -04:00
Josh Brower
8c16feb772 Rename Fleet pipelines 2023-07-09 12:22:55 -04:00
Josh Brower
ff3bb11fbb Elastic Fleet Certs Refactor 2023-07-07 16:44:16 -04:00
Josh Brower
7d0251952c Filter out uneeded Logstash metadata 2023-05-17 11:06:16 -04:00
Josh Brower
24445cf36a Rename Fleet pipelines 2023-05-16 16:43:21 -04:00
m0duspwnens
082704ce1f logstash jinja for ui 2023-05-04 13:07:07 -04:00
Mike Reeves
3d10a60502 Fix annotations and defaults for logstash 2023-05-03 10:01:44 -04:00
Wes
d823d5dcc9 Rename @metadata to metadata to ensure it's not lost between Logstash pipelines 2023-04-19 20:17:10 +00:00
Josh Brower
1944d09978 Logstash certs fixup 2023-04-17 11:34:57 -04:00
Josh Brower
b8d8a5fd6b Remove default outputs 2023-01-31 17:02:41 -05:00
Josh Brower
18a54b86f4 More fixes 2023-01-31 14:57:39 -05:00
Wes
e4271043c6 Remove unnecessary Logstash pipelines 2023-01-26 18:05:14 +00:00
Wes
44d149b1c3 Allow imported data to use a tag of 'import' 2023-01-24 17:01:52 +00:00
Mike Reeves
66924b63a7 Update 9999_output_redis.conf.jinja 2023-01-11 14:53:16 -05:00
Mike Reeves
bdaed849ea Update 0900_input_redis.conf.jinja 2023-01-11 14:52:32 -05:00
Wes
5d86edeed4 Modify Logstash Elastic Agent output to accomodate for events with and without 'metadata.pipeline' 2023-01-11 13:57:32 +00:00
Mike Reeves
831300b540 Require password auth for redis access 2023-01-04 11:02:40 -05:00
m0duspwnens
b526532ab6 use global vars in states 2022-10-11 11:57:15 -04:00
Wes
1a90eeb1b1 Remove Osquery live query Logstash output configuration 2022-09-15 19:45:28 +00:00
Wes
926a1e0189 Remove Snort output configuration 2022-09-14 14:22:00 +00:00
Wes
ce3ea456b6 Remove flow output configuration 2022-09-14 14:21:21 +00:00
Wes
d1a8b88eb9 Remove postprocess configuration 2022-09-14 14:20:24 +00:00
Wes
e3cd8a9c6a Remove main pipeline configuration 2022-09-14 14:20:08 +00:00
Wes
43f89adbd4 Remove preprocess configuration 2022-09-14 14:19:07 +00:00
Mike Reeves
2bd9dd80e2 Move In Day 2022-09-07 09:06:25 -04:00
Wes Lambert
26698cfd07 Add Logstash output for dedicated Kratos index 2022-07-08 15:55:55 +00:00
Wes Lambert
f613d8ad86 Add RITA Logstash config 2022-03-22 17:36:18 +00:00
Wes Lambert
9db1510b0e Initial composable template configuration and base mappings 2022-02-02 02:08:31 +00:00
m0duspwnens
2e4ed8062e simplify wazuh agent ip logic 2021-12-16 11:11:01 -05:00
m0duspwnens
d0b0970353 Merge remote-tracking branch 'remotes/origin/dev' into issue/6469 2021-12-15 17:08:56 -05:00
m0duspwnens
0c6aba16ec fix redis input 2021-12-14 23:42:37 -05:00
m0duspwnens
15b8d80b71 fix host for input_redis 2021-12-14 18:51:43 -05:00
m0duspwnens
55b74abcc5 extra_hosts and redis_input for logstash 2021-12-14 18:49:30 -05:00