CSEC_PUBLIC/securityonion
1
0
Fork 0
mirror of https://github.com/Security-Onion-Solutions/securityonion.git synced 2025-12-06 09:12:45 +01:00
Code Issues Packages Projects Releases Wiki Activity
14,820 Commits 24 Branches 119 Tags
093cbc5ebcfc59f70f9e6860006e802ce61fa7fc
Commit Graph

14820 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
reyesj2
093cbc5ebc Reconfigure Kafka defaults 
- Set default number of partitions per topic -> 3. Helps ensure that out of the box we can take advantage of multi-node Kafka clusters via load balancing across atleast 3 brokers. Also multiple searchnodes will be able to pull from each topic. In this case 3 searchnodes (consumers) would be able to pull from all topics concurrently.
- Set default replication factor -> 2. This is the minimum value required for redundancy. Every partition will have 1 replica. In this case if we have 2 brokers each topic will have 3 partitions (load balanced across brokers) and each partition will have a replica on separate broker for redundancy

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 15:10:13 -04:00
reyesj2
f663ef8c16 Setup Kafka to use PKCS12 and remove need for converting to JKS 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 14:53:28 -04:00
reyesj2
de9f6425f9 Automatically switch between Kafka output policy and logstash output policy when globals.pipeline changes 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 12:13:46 -04:00
reyesj2
47ced60243 Create new Kafka output policy using salt 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 14:49:51 -04:00
reyesj2
58ebbfba20 Add kafka state to standalone highstate 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 13:03:14 -04:00
reyesj2
e164d15ec6 Generate different Kafka certs for different SO nodetypes 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 13:02:47 -04:00
reyesj2
3efdb4e532 Reconfigure logstash Kafka input 
- TODO: Configure what topics are pulled to searchnodes via the SOC UI

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 13:01:29 -04:00
reyesj2
de0af58cf8 Write out Kafka pillar path 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 10:45:46 -04:00
reyesj2
84abfa6881 Remove check for existing value since Kafka pillar is made empty on upgrade 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 10:45:05 -04:00
reyesj2
6b60e85a33 Make kafka configuration changes prior to 2.4.70 upgrade 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 10:15:26 -04:00
reyesj2
63f3e23e2b soup typo 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 09:54:19 -04:00
reyesj2
eb1249618b Update soup for Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 09:27:01 -04:00
reyesj2
cef9bb1487 Dynamically create Kafka topics based on event.module from elastic agent logs eg. zeek-topic. Depends on Kafka brokers having auto.create.topics.enable set to true 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 09:16:13 -04:00
reyesj2
bb49944b96 Setup elastic fleet rollover from logstash -> kafka output policy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-30 16:47:40 -04:00
reyesj2
fcc4050f86 Add id to grid-kafka fleet output policy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-30 12:59:53 -04:00
reyesj2
9c83a52c6d Add Kafka output to elastic-fleet setup. Includes separating topics by event.module with fallback to default-logs if no event.module is specified or doesn't match processors 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-30 12:01:31 -04:00
reyesj2
a6e8b25969 Add Kafka connectivity between manager - > receiver nodes. 
Add connectivity to Kafka between other node types that may need to publish to Kafka.

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 15:48:57 -04:00
reyesj2
529bc01d69 Add missing configuration for nodes running Kafka broker role only 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:53:52 -04:00
reyesj2
11055b1d32 Rename kafkapass -> kafka_pass 
Run so-kafka-clusterid within nodes.sls state so switchover is consistent

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:09:09 -04:00
reyesj2
fd9a91420d Use SOC UI to configure list of KRaft (Kafka) controllers for cluster 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:37:24 -04:00
reyesj2
529c8d7cf2 Remove salt reactor for Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:35:46 -04:00
reyesj2
086ebe1a7c Split kafka defaults between broker / controller 
Setup config.map.jinja to update broker / controller / combined node types

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:08:14 -04:00
reyesj2
29c964cca1 Set kafka.nodes state to run first to populate kafka.nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:04:52 -04:00
reyesj2
36573d6005 Update kafka cert permissions 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-23 16:45:36 -04:00
reyesj2
aa0c589361 Update kafka managed node pillar template to include its process.role 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-23 13:51:12 -04:00
reyesj2
685b80e519 Merge remote-tracking branch 'remotes/origin/kaffytaffy' into reyesj2/kafka 2024-04-22 16:45:59 -04:00
reyesj2
5a401af1fd Update kafka process_x_roles annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:44:35 -04:00
reyesj2
25d63f7516 Setup kafka reactor for managing kafka controllers globally 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:42:59 -04:00
m0duspwnens
6c5e0579cf logging changes. ensure salt master has pillarWatch engine 2024-04-19 09:32:32 -04:00
reyesj2
4ac04a1a46 add kafkapass soc annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 16:46:36 -04:00
reyesj2
746128e37b update so-kafka-clusterid 
This is a temporary script used to setup kafka secret and clusterid needed for kafka to start. This scripts functionality will be replaced by soup/setup scripts

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:13:29 -04:00
reyesj2
fe81ffaf78 Variables no longer used. Replaced by map file 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:11:22 -04:00
m0duspwnens
1f6eb9cdc3 match keys better. go through files reverse first found is prio 2024-04-18 13:50:37 -04:00
reyesj2
5cc358de4e Update map files to handle empty kafka:nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 11:58:25 -04:00
m0duspwnens
610dd2c08d improve it 2024-04-18 11:11:14 -04:00
m0duspwnens
506bbd314d more comments, better logging 2024-04-18 10:26:10 -04:00
m0duspwnens
4caa6a10b5 watch a pillar in files and take action 2024-04-17 18:09:04 -04:00
reyesj2
665b7197a6 Update Kafka nodeid 
Update so-minion to include running kafka.nodes state to ensure nodeid is generated for new brokers

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-17 17:08:41 -04:00
m0duspwnens
4b79623ce3 watch pillar files for changes and do something 2024-04-16 16:51:35 -04:00
m0duspwnens
c4994a208b restart salt minion if a manager and signing policies change 2024-04-15 11:37:21 -04:00
reyesj2
eedea2ca88 Merge remote-tracking branch 'remotes/origin/kaffytaffy' into reyesj2/kafka 2024-04-12 16:24:33 -04:00
reyesj2
de6ea29e3b update default process.role to broker only 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 16:18:53 -04:00
m0duspwnens
bb983d4ba2 just broker as default process 2024-04-12 16:16:03 -04:00
m0duspwnens
c014508519 need /opt/so/conf/ca/cacerts on receiver for kafka to run 2024-04-12 13:50:25 -04:00
reyesj2
fcfbb1e857 Merge kaffytaffy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 12:50:56 -04:00
reyesj2
911ee579a9 Typo 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 12:16:20 -04:00
reyesj2
a6ff92b099 Note to remove so-kafka-clusterid. Update soup and setup to generate needed kafka pillar values 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 12:11:18 -04:00
m0duspwnens
d73ba7dd3e order kafka pillar assignment 2024-04-12 11:55:26 -04:00
m0duspwnens
04ddcd5c93 add receiver managersearch and standalone to kafka.nodes pillar 2024-04-12 11:52:57 -04:00
reyesj2
af29ae1968 Merge kaffytaffy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 11:43:46 -04:00