Commit Graph

46 Commits

Author SHA1 Message Date
reyesj2
529bc01d69 Add missing configuration for nodes running Kafka broker role only
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-29 14:53:52 -04:00
reyesj2
11055b1d32 Rename kafkapass -> kafka_pass
Run so-kafka-clusterid within nodes.sls state so switchover is consistent

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-29 14:09:09 -04:00
reyesj2
fd9a91420d Use SOC UI to configure list of KRaft (Kafka) controllers for cluster
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-29 11:37:24 -04:00
reyesj2
529c8d7cf2 Remove salt reactor for Kafka
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-29 11:35:46 -04:00
reyesj2
086ebe1a7c Split kafka defaults between broker / controller
Setup config.map.jinja to update broker / controller / combined node types

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-29 09:08:14 -04:00
reyesj2
29c964cca1 Set kafka.nodes state to run first to populate kafka.nodes pillar
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-29 09:04:52 -04:00
reyesj2
aa0c589361 Update kafka managed node pillar template to include its process.role
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-23 13:51:12 -04:00
reyesj2
5a401af1fd Update kafka process_x_roles annotation
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-22 16:44:35 -04:00
reyesj2
25d63f7516 Setup kafka reactor for managing kafka controllers globally
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-22 16:42:59 -04:00
reyesj2
4ac04a1a46 add kafkapass soc annotation
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-18 16:46:36 -04:00
reyesj2
746128e37b update so-kafka-clusterid
This is a temporary script used to setup kafka secret and clusterid needed for kafka to start. This scripts functionality will be replaced by soup/setup scripts

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-18 15:13:29 -04:00
reyesj2
fe81ffaf78 Variables no longer used. Replaced by map file
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-18 15:11:22 -04:00
reyesj2
5cc358de4e Update map files to handle empty kafka:nodes pillar
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-18 11:58:25 -04:00
reyesj2
665b7197a6 Update Kafka nodeid
Update so-minion to include running kafka.nodes state to ensure nodeid is generated for new brokers

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-17 17:08:41 -04:00
m0duspwnens
bb983d4ba2 just broker as default process 2024-04-12 16:16:03 -04:00
m0duspwnens
c014508519 need /opt/so/conf/ca/cacerts on receiver for kafka to run 2024-04-12 13:50:25 -04:00
reyesj2
911ee579a9 Typo
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-12 12:16:20 -04:00
reyesj2
af29ae1968 Merge kaffytaffy
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-12 11:43:46 -04:00
m0duspwnens
a54a72c269 move kafka_cluster_id to kafka:cluster_id 2024-04-12 11:19:20 -04:00
reyesj2
3955587372 Use global.pipeline for redis / kafka states
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-11 16:20:09 -04:00
reyesj2
6b28dc72e8 Update annotation for global.pipeline
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-11 15:38:33 -04:00
reyesj2
af53dcda1b Remove references to kafkanode
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-11 15:32:00 -04:00
m0duspwnens
d3bd56b131 disable logstash and redis if kafka enabled 2024-04-10 14:13:27 -04:00
m0duspwnens
86b984001d annotations and enable/disable from ui 2024-04-10 10:39:06 -04:00
m0duspwnens
bd5fe43285 jinja config files 2024-04-09 11:07:53 -04:00
m0duspwnens
d38051e806 fix client and server properties formatting 2024-04-09 10:36:37 -04:00
m0duspwnens
daa5342986 items not keys in for loop 2024-04-09 10:22:05 -04:00
m0duspwnens
c48436ccbf fix dict update 2024-04-09 10:19:17 -04:00
m0duspwnens
7aa00faa6c fix var 2024-04-09 09:31:54 -04:00
m0duspwnens
6217a7b9a9 add defaults and jijafy kafka config 2024-04-09 09:27:21 -04:00
reyesj2
433309ef1a Generate kafka cluster id if it doesn't exist
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-05 09:35:12 -04:00
reyesj2
735cfb4c29 Autogenerate kafka topics when a message it sent to non-existing topic
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-04 16:45:58 -04:00
reyesj2
6202090836 Merge remote-tracking branch 'origin/kaffytaffy' into reyesj2/kafka 2024-04-04 16:27:06 -04:00
reyesj2
40b08d737c Generate kafka keystore on changes to kafka.key
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-04 16:16:53 -04:00
m0duspwnens
4c5b42b898 restart container on server config changes 2024-04-04 15:47:01 -04:00
reyesj2
dc27bbb01d Set kafka heap size. To be later configured from SOC
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-03 14:30:52 -04:00
m0duspwnens
b863060df1 kafka broker and listener on 0.0.0.0 2024-04-03 11:05:24 -04:00
m0duspwnens
c3f02a698e add kafka nodes as extra hosts for the container 2024-04-03 10:23:36 -04:00
m0duspwnens
db106f8ca1 listen on 0.0.0.0 for CONTROLLER 2024-04-03 10:22:47 -04:00
m0duspwnens
639bf05081 add so-manager to kafka.nodes pillar 2024-04-03 08:52:26 -04:00
reyesj2
82830c8173 Fix typos and fix error related to elasticsearch saltstate being called from logstash state. Logstash will be removed from kafkanodes in future
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-02 16:37:39 -04:00
reyesj2
7f5741c43b Fix kafka storage setup
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-02 16:36:22 -04:00
reyesj2
643d4831c1 CRLF -> LF
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-02 16:35:14 -04:00
reyesj2
b032eed22a Update kafka to use manager docker registry
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-02 16:34:06 -04:00
reyesj2
1b49c8540e Fix kafka keystore script
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2024-04-02 16:32:15 -04:00
reyesj2
86dc7cc804 Kafka init
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
2023-11-29 13:34:25 -05:00