CSEC_PUBLIC/securityonion
1
0
Fork 0
mirror of https://github.com/Security-Onion-Solutions/securityonion.git synced 2025-12-06 17:22:49 +01:00
Code Issues Packages Projects Releases Wiki Activity
15,939 Commits 24 Branches 119 Tags
e02cb30f1b262c53afa3cce11d0b47ff07b90caa
Commit Graph

78 Commits

Author SHA1 Message Date
reyesj2
cf1335dd84 searchnode logstash-kafka cert generation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-20 11:31:33 -04:00
reyesj2
5d322ebc0b Allow searchnodes to run kafka.ssl state for kafka-logstash cert generation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-12 14:45:11 -04:00
reyesj2
4182ff66a0 rearrange kafka pillar, declutters SOC ui 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-11 16:37:16 -04:00
reyesj2
4a88dedcb8 Fixin kafka.ssl state and include name for kafka_user 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-10 16:18:46 -04:00
reyesj2
d5faf535c3 Only interact with logstash configuration when Kafka pipeline is enabled otherwise leave it default 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-10 11:36:44 -04:00
reyesj2
8e1edd1d91 split Kafka ssl from ssl/init. Certs won't be generated until Kafka is enabled. Also runs some clean up for old Kafka certs 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-10 11:32:43 -04:00
reyesj2
d791b23838 Generate new Kafka truststore 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-07-10 11:29:09 -04:00
reyesj2
81ce762250 delete commented block 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-24 14:06:48 -04:00
reyesj2
cb727bf48d remove unused sbin_jinja from kafka config 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-24 13:45:13 -04:00
reyesj2
2eea671857 more precise wording in kafka annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-20 16:16:55 -04:00
reyesj2
73acfbf864 update kafka annotations 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-20 16:02:45 -04:00
reyesj2
4d1f2c2bc1 fix kafka elastic fleet output policy setup 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-14 23:04:08 -04:00
reyesj2
4e50dabc56 refix typos 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-14 23:03:06 -04:00
reyesj2
8f8ece2b34 Only comment out so-kafka from so-status when it exists & only run ensure_default_pipeline when Kafka is configured 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-13 15:50:34 -04:00
reyesj2
19bfd5beca fix kafka nodeid assignment to increment correctly 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-13 12:16:39 -04:00
reyesj2
f372b0907b Use kafka:password for kafka certs 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 15:41:10 -04:00
reyesj2
e8106befe9 Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 12:05:16 -04:00
reyesj2
b7eebad2a5 Update Kafka self reset & add initial Kafka wrapper scripts to build out 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 11:01:40 -04:00
reyesj2
628893fd5b remove redundant 'kafka_' from annotations & defaults 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-11 11:56:21 -04:00
reyesj2
ca7b89c308 Added Kafka reset to SOC UI. Incase of changing an active broker to a controller topics may become unavailable. Resolving this would require manual intervention. This option allows running a reset to start from a clean slate to then configure cluster to desired state before reenabling Kafka as global pipeline. 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-11 11:21:13 -04:00
reyesj2
284c1be85f Update Kafka controller(s) via SOC UI 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-10 11:08:54 -04:00
reyesj2
fb1d4fdd3c update license 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-04 12:33:51 -04:00
reyesj2
2e85a28c02 Remove so-kafka-clusterid script, created during soup 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-02 18:25:59 -04:00
reyesj2
dbb99d0367 Remove bad config 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-30 15:10:15 -04:00
reyesj2
386be4e746 WIP: Manage Kafka nodes pillar role value 
This way when kafka_controllers is updated the pillar value gets updated and any non-controllers get updated to revert to 'broker' only role.
 Needs more testing when a new controller joins in this manner Kafka errors due to cluster metadata being out of sync. One solution is to remove /nsm/kafka/data/__cluster_metadata-0/quorum-state and restart cluster. Alternative is working with Kafka cli tools to inform cluster of new voter, likely best option but requires a wrapper script of some sort to be created for updating cluster in-place.
Easiest option is to have all receivers join grid and then configure Kafka with specific controllers via SOC UI prior to enabling Kafka. This way Kafka cluster comes up in the desired configuration with no need for immediately modifying cluster

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-29 16:48:39 -04:00
reyesj2
d9ec556061 Update some annotations and defaults 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-29 16:41:02 -04:00
reyesj2
382cd24a57 Small changes needed for using new Kafka docker image + added Kafka logging output to /opt/so/log/kafka/ 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-22 13:39:21 -04:00
reyesj2
91f8b1fef7 Set default replication factor back to Kafka default 
If replication factor is > 1 Kafka will fail to start until another broker is added
  - For internal automated testing purposes a Standalone will be utilized

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-22 13:35:09 -04:00
reyesj2
dff609d829 Add basic read-only metric collection from Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-08 16:13:09 -04:00
reyesj2
093cbc5ebc Reconfigure Kafka defaults 
- Set default number of partitions per topic -> 3. Helps ensure that out of the box we can take advantage of multi-node Kafka clusters via load balancing across atleast 3 brokers. Also multiple searchnodes will be able to pull from each topic. In this case 3 searchnodes (consumers) would be able to pull from all topics concurrently.
- Set default replication factor -> 2. This is the minimum value required for redundancy. Every partition will have 1 replica. In this case if we have 2 brokers each topic will have 3 partitions (load balanced across brokers) and each partition will have a replica on separate broker for redundancy

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 15:10:13 -04:00
reyesj2
f663ef8c16 Setup Kafka to use PKCS12 and remove need for converting to JKS 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 14:53:28 -04:00
reyesj2
47ced60243 Create new Kafka output policy using salt 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 14:49:51 -04:00
reyesj2
529bc01d69 Add missing configuration for nodes running Kafka broker role only 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:53:52 -04:00
reyesj2
11055b1d32 Rename kafkapass -> kafka_pass 
Run so-kafka-clusterid within nodes.sls state so switchover is consistent

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:09:09 -04:00
reyesj2
fd9a91420d Use SOC UI to configure list of KRaft (Kafka) controllers for cluster 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:37:24 -04:00
reyesj2
529c8d7cf2 Remove salt reactor for Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:35:46 -04:00
reyesj2
086ebe1a7c Split kafka defaults between broker / controller 
Setup config.map.jinja to update broker / controller / combined node types

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:08:14 -04:00
reyesj2
29c964cca1 Set kafka.nodes state to run first to populate kafka.nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:04:52 -04:00
reyesj2
aa0c589361 Update kafka managed node pillar template to include its process.role 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-23 13:51:12 -04:00
reyesj2
5a401af1fd Update kafka process_x_roles annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:44:35 -04:00
reyesj2
25d63f7516 Setup kafka reactor for managing kafka controllers globally 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:42:59 -04:00
reyesj2
4ac04a1a46 add kafkapass soc annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 16:46:36 -04:00
reyesj2
746128e37b update so-kafka-clusterid 
This is a temporary script used to setup kafka secret and clusterid needed for kafka to start. This scripts functionality will be replaced by soup/setup scripts

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:13:29 -04:00
reyesj2
fe81ffaf78 Variables no longer used. Replaced by map file 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:11:22 -04:00
reyesj2
5cc358de4e Update map files to handle empty kafka:nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 11:58:25 -04:00
reyesj2
665b7197a6 Update Kafka nodeid 
Update so-minion to include running kafka.nodes state to ensure nodeid is generated for new brokers

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-17 17:08:41 -04:00
m0duspwnens
bb983d4ba2 just broker as default process 2024-04-12 16:16:03 -04:00
m0duspwnens
c014508519 need /opt/so/conf/ca/cacerts on receiver for kafka to run 2024-04-12 13:50:25 -04:00
reyesj2
911ee579a9 Typo 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 12:16:20 -04:00
reyesj2
af29ae1968 Merge kaffytaffy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 11:43:46 -04:00