CSEC_PUBLIC/securityonion
1
0
Fork 0
mirror of https://github.com/Security-Onion-Solutions/securityonion.git synced 2025-12-07 17:52:46 +01:00
Code Issues Packages Projects Releases Wiki Activity
15,250 Commits 24 Branches 119 Tags
2bc53f9868ce7b8afab6cb525e735c0be4cc61ce
Commit Graph

62 Commits

Author SHA1 Message Date
reyesj2
e8106befe9 Append '-securityonion' to all Security Onion related Kafka topics. Adjust logstash to ingest all topics ending in '-securityonion' to avoid having to manually list topic names 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 12:05:16 -04:00
reyesj2
b7eebad2a5 Update Kafka self reset & add initial Kafka wrapper scripts to build out 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-12 11:01:40 -04:00
reyesj2
628893fd5b remove redundant 'kafka_' from annotations & defaults 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-11 11:56:21 -04:00
reyesj2
ca7b89c308 Added Kafka reset to SOC UI. Incase of changing an active broker to a controller topics may become unavailable. Resolving this would require manual intervention. This option allows running a reset to start from a clean slate to then configure cluster to desired state before reenabling Kafka as global pipeline. 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-11 11:21:13 -04:00
reyesj2
284c1be85f Update Kafka controller(s) via SOC UI 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-10 11:08:54 -04:00
reyesj2
fb1d4fdd3c update license 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-04 12:33:51 -04:00
reyesj2
2e85a28c02 Remove so-kafka-clusterid script, created during soup 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-06-02 18:25:59 -04:00
reyesj2
dbb99d0367 Remove bad config 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-30 15:10:15 -04:00
reyesj2
386be4e746 WIP: Manage Kafka nodes pillar role value 
This way when kafka_controllers is updated the pillar value gets updated and any non-controllers get updated to revert to 'broker' only role.
 Needs more testing when a new controller joins in this manner Kafka errors due to cluster metadata being out of sync. One solution is to remove /nsm/kafka/data/__cluster_metadata-0/quorum-state and restart cluster. Alternative is working with Kafka cli tools to inform cluster of new voter, likely best option but requires a wrapper script of some sort to be created for updating cluster in-place.
Easiest option is to have all receivers join grid and then configure Kafka with specific controllers via SOC UI prior to enabling Kafka. This way Kafka cluster comes up in the desired configuration with no need for immediately modifying cluster

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-29 16:48:39 -04:00
reyesj2
d9ec556061 Update some annotations and defaults 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-29 16:41:02 -04:00
reyesj2
382cd24a57 Small changes needed for using new Kafka docker image + added Kafka logging output to /opt/so/log/kafka/ 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-22 13:39:21 -04:00
reyesj2
91f8b1fef7 Set default replication factor back to Kafka default 
If replication factor is > 1 Kafka will fail to start until another broker is added
  - For internal automated testing purposes a Standalone will be utilized

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-22 13:35:09 -04:00
reyesj2
dff609d829 Add basic read-only metric collection from Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-08 16:13:09 -04:00
reyesj2
093cbc5ebc Reconfigure Kafka defaults 
- Set default number of partitions per topic -> 3. Helps ensure that out of the box we can take advantage of multi-node Kafka clusters via load balancing across atleast 3 brokers. Also multiple searchnodes will be able to pull from each topic. In this case 3 searchnodes (consumers) would be able to pull from all topics concurrently.
- Set default replication factor -> 2. This is the minimum value required for redundancy. Every partition will have 1 replica. In this case if we have 2 brokers each topic will have 3 partitions (load balanced across brokers) and each partition will have a replica on separate broker for redundancy

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 15:10:13 -04:00
reyesj2
f663ef8c16 Setup Kafka to use PKCS12 and remove need for converting to JKS 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-02 14:53:28 -04:00
reyesj2
47ced60243 Create new Kafka output policy using salt 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-05-01 14:49:51 -04:00
reyesj2
529bc01d69 Add missing configuration for nodes running Kafka broker role only 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:53:52 -04:00
reyesj2
11055b1d32 Rename kafkapass -> kafka_pass 
Run so-kafka-clusterid within nodes.sls state so switchover is consistent

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 14:09:09 -04:00
reyesj2
fd9a91420d Use SOC UI to configure list of KRaft (Kafka) controllers for cluster 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:37:24 -04:00
reyesj2
529c8d7cf2 Remove salt reactor for Kafka 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 11:35:46 -04:00
reyesj2
086ebe1a7c Split kafka defaults between broker / controller 
Setup config.map.jinja to update broker / controller / combined node types

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:08:14 -04:00
reyesj2
29c964cca1 Set kafka.nodes state to run first to populate kafka.nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-29 09:04:52 -04:00
reyesj2
aa0c589361 Update kafka managed node pillar template to include its process.role 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-23 13:51:12 -04:00
reyesj2
5a401af1fd Update kafka process_x_roles annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:44:35 -04:00
reyesj2
25d63f7516 Setup kafka reactor for managing kafka controllers globally 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-22 16:42:59 -04:00
reyesj2
4ac04a1a46 add kafkapass soc annotation 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 16:46:36 -04:00
reyesj2
746128e37b update so-kafka-clusterid 
This is a temporary script used to setup kafka secret and clusterid needed for kafka to start. This scripts functionality will be replaced by soup/setup scripts

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:13:29 -04:00
reyesj2
fe81ffaf78 Variables no longer used. Replaced by map file 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 15:11:22 -04:00
reyesj2
5cc358de4e Update map files to handle empty kafka:nodes pillar 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-18 11:58:25 -04:00
reyesj2
665b7197a6 Update Kafka nodeid 
Update so-minion to include running kafka.nodes state to ensure nodeid is generated for new brokers

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-17 17:08:41 -04:00
m0duspwnens
bb983d4ba2 just broker as default process 2024-04-12 16:16:03 -04:00
m0duspwnens
c014508519 need /opt/so/conf/ca/cacerts on receiver for kafka to run 2024-04-12 13:50:25 -04:00
reyesj2
911ee579a9 Typo 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 12:16:20 -04:00
reyesj2
af29ae1968 Merge kaffytaffy 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-12 11:43:46 -04:00
m0duspwnens
a54a72c269 move kafka_cluster_id to kafka:cluster_id 2024-04-12 11:19:20 -04:00
reyesj2
3955587372 Use global.pipeline for redis / kafka states 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-11 16:20:09 -04:00
reyesj2
6b28dc72e8 Update annotation for global.pipeline 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-11 15:38:33 -04:00
reyesj2
af53dcda1b Remove references to kafkanode 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-11 15:32:00 -04:00
m0duspwnens
d3bd56b131 disable logstash and redis if kafka enabled 2024-04-10 14:13:27 -04:00
m0duspwnens
86b984001d annotations and enable/disable from ui 2024-04-10 10:39:06 -04:00
m0duspwnens
bd5fe43285 jinja config files 2024-04-09 11:07:53 -04:00
m0duspwnens
d38051e806 fix client and server properties formatting 2024-04-09 10:36:37 -04:00
m0duspwnens
daa5342986 items not keys in for loop 2024-04-09 10:22:05 -04:00
m0duspwnens
c48436ccbf fix dict update 2024-04-09 10:19:17 -04:00
m0duspwnens
7aa00faa6c fix var 2024-04-09 09:31:54 -04:00
m0duspwnens
6217a7b9a9 add defaults and jijafy kafka config 2024-04-09 09:27:21 -04:00
reyesj2
433309ef1a Generate kafka cluster id if it doesn't exist 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-05 09:35:12 -04:00
reyesj2
735cfb4c29 Autogenerate kafka topics when a message it sent to non-existing topic 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-04 16:45:58 -04:00
reyesj2
6202090836 Merge remote-tracking branch 'origin/kaffytaffy' into reyesj2/kafka 2024-04-04 16:27:06 -04:00
reyesj2
40b08d737c Generate kafka keystore on changes to kafka.key 
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
 2024-04-04 16:16:53 -04:00