CSEC_PUBLIC/securityonion
1
0
Fork 0
mirror of https://github.com/Security-Onion-Solutions/securityonion.git synced 2026-04-25 05:57:49 +02:00
Code Issues Packages Projects Releases Wiki Activity
15,264 Commits 40 Branches 125 Tags
a0030b27e231fc0bc9411979516e22799e2fd6cd
Commit Graph

65 Commits

Author SHA1 Message Date
reyesj2 8f8ece2b34 Only comment out so-kafka from so-status when it exists & only run ensure_default_pipeline when Kafka is configured 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-13 15:50:34 -04:00
reyesj2 19bfd5beca fix kafka nodeid assignment to increment correctly 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-13 12:16:39 -04:00
reyesj2 f372b0907b Use kafka:password for kafka certs 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 15:41:10 -04:00
reyesj2 e8106befe9 Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 12:05:16 -04:00
reyesj2 b7eebad2a5 Update Kafka self reset & add initial Kafka wrapper scripts to build out 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 11:01:40 -04:00
reyesj2 628893fd5b remove redundant 'kafka_' from annotations & defaults 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-11 11:56:21 -04:00
reyesj2 ca7b89c308 Added Kafka reset to SOC UI. Incase of changing an active broker to a controller topics may become unavailable. Resolving this would require manual intervention. This option allows running a reset to start from a clean slate to then configure cluster to desired state before reenabling Kafka as global pipeline. 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-11 11:21:13 -04:00
reyesj2 284c1be85f Update Kafka controller(s) via SOC UI 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-10 11:08:54 -04:00
reyesj2 fb1d4fdd3c update license 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-04 12:33:51 -04:00
reyesj2 2e85a28c02 Remove so-kafka-clusterid script, created during soup 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-02 18:25:59 -04:00
reyesj2 dbb99d0367 Remove bad config 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-30 15:10:15 -04:00
reyesj2 386be4e746 WIP: Manage Kafka nodes pillar role value 
This way when kafka_controllers is updated the pillar value gets updated and any non-controllers get updated to revert to 'broker' only role.
 Needs more testing when a new controller joins in this manner Kafka errors due to cluster metadata being out of sync. One solution is to remove /nsm/kafka/data/__cluster_metadata-0/quorum-state and restart cluster. Alternative is working with Kafka cli tools to inform cluster of new voter, likely best option but requires a wrapper script of some sort to be created for updating cluster in-place.
Easiest option is to have all receivers join grid and then configure Kafka with specific controllers via SOC UI prior to enabling Kafka. This way Kafka cluster comes up in the desired configuration with no need for immediately modifying cluster

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-29 16:48:39 -04:00
reyesj2 d9ec556061 Update some annotations and defaults 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-29 16:41:02 -04:00
reyesj2 382cd24a57 Small changes needed for using new Kafka docker image + added Kafka logging output to /opt/so/log/kafka/ 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-22 13:39:21 -04:00
reyesj2 91f8b1fef7 Set default replication factor back to Kafka default 
If replication factor is > 1 Kafka will fail to start until another broker is added
  - For internal automated testing purposes a Standalone will be utilized

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-22 13:35:09 -04:00
reyesj2 dff609d829 Add basic read-only metric collection from Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-08 16:13:09 -04:00
reyesj2 093cbc5ebc Reconfigure Kafka defaults 
- Set default number of partitions per topic -> 3. Helps ensure that out of the box we can take advantage of multi-node Kafka clusters via load balancing across atleast 3 brokers. Also multiple searchnodes will be able to pull from each topic. In this case 3 searchnodes (consumers) would be able to pull from all topics concurrently.
- Set default replication factor -> 2. This is the minimum value required for redundancy. Every partition will have 1 replica. In this case if we have 2 brokers each topic will have 3 partitions (load balanced across brokers) and each partition will have a replica on separate broker for redundancy

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 15:10:13 -04:00
reyesj2 f663ef8c16 Setup Kafka to use PKCS12 and remove need for converting to JKS 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 14:53:28 -04:00
reyesj2 47ced60243 Create new Kafka output policy using salt 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 14:49:51 -04:00
reyesj2 529bc01d69 Add missing configuration for nodes running Kafka broker role only 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:53:52 -04:00
reyesj2 11055b1d32 Rename kafkapass -> kafka_pass 
Run so-kafka-clusterid within nodes.sls state so switchover is consistent

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:09:09 -04:00
reyesj2 fd9a91420d Use SOC UI to configure list of KRaft (Kafka) controllers for cluster 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:37:24 -04:00
reyesj2 529c8d7cf2 Remove salt reactor for Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:35:46 -04:00
reyesj2 086ebe1a7c Split kafka defaults between broker / controller 
Setup config.map.jinja to update broker / controller / combined node types

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:08:14 -04:00
reyesj2 29c964cca1 Set kafka.nodes state to run first to populate kafka.nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:04:52 -04:00
reyesj2 aa0c589361 Update kafka managed node pillar template to include its process.role 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-23 13:51:12 -04:00
reyesj2 5a401af1fd Update kafka process_x_roles annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:44:35 -04:00
reyesj2 25d63f7516 Setup kafka reactor for managing kafka controllers globally 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:42:59 -04:00
reyesj2 4ac04a1a46 add kafkapass soc annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 16:46:36 -04:00
reyesj2 746128e37b update so-kafka-clusterid 
This is a temporary script used to setup kafka secret and clusterid needed for kafka to start. This scripts functionality will be replaced by soup/setup scripts

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:13:29 -04:00
reyesj2 fe81ffaf78 Variables no longer used. Replaced by map file 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:11:22 -04:00
reyesj2 5cc358de4e Update map files to handle empty kafka:nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 11:58:25 -04:00
reyesj2 665b7197a6 Update Kafka nodeid 
Update so-minion to include running kafka.nodes state to ensure nodeid is generated for new brokers

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-17 17:08:41 -04:00
m0duspwnens bb983d4ba2 just broker as default process 2024-04-12 16:16:03 -04:00
m0duspwnens c014508519 need /opt/so/conf/ca/cacerts on receiver for kafka to run 2024-04-12 13:50:25 -04:00
reyesj2 911ee579a9 Typo 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 12:16:20 -04:00
reyesj2 af29ae1968 Merge kaffytaffy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 11:43:46 -04:00
m0duspwnens a54a72c269 move kafka_cluster_id to kafka:cluster_id 2024-04-12 11:19:20 -04:00
reyesj2 3955587372 Use global.pipeline for redis / kafka states 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-11 16:20:09 -04:00
reyesj2 6b28dc72e8 Update annotation for global.pipeline 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-11 15:38:33 -04:00
reyesj2 af53dcda1b Remove references to kafkanode 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-11 15:32:00 -04:00
m0duspwnens d3bd56b131 disable logstash and redis if kafka enabled 2024-04-10 14:13:27 -04:00
m0duspwnens 86b984001d annotations and enable/disable from ui 2024-04-10 10:39:06 -04:00
m0duspwnens bd5fe43285 jinja config files 2024-04-09 11:07:53 -04:00
m0duspwnens d38051e806 fix client and server properties formatting 2024-04-09 10:36:37 -04:00
m0duspwnens daa5342986 items not keys in for loop 2024-04-09 10:22:05 -04:00
m0duspwnens c48436ccbf fix dict update 2024-04-09 10:19:17 -04:00
m0duspwnens 7aa00faa6c fix var 2024-04-09 09:31:54 -04:00
m0duspwnens 6217a7b9a9 add defaults and jijafy kafka config 2024-04-09 09:27:21 -04:00
reyesj2 433309ef1a Generate kafka cluster id if it doesn't exist 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-05 09:35:12 -04:00