Reconfigure Kafka defaults

- Set default number of partitions per topic -> 3. Helps ensure that out of the box we can take advantage of multi-node Kafka clusters via load balancing across atleast 3 brokers. Also multiple searchnodes will be able to pull from each topic. In this case 3 searchnodes (consumers) would be able to pull from all topics concurrently.
- Set default replication factor -> 2. This is the minimum value required for redundancy. Every partition will have 1 replica. In this case if we have 2 brokers each topic will have 3 partitions (load balanced across brokers) and each partition will have a replica on separate broker for redundancy

Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com>
This commit is contained in:
reyesj2
2024-05-02 15:10:13 -04:00
parent f663ef8c16
commit 093cbc5ebc
2 changed files with 8 additions and 2 deletions

View File

@@ -8,6 +8,7 @@ kafka:
advertised_x_listeners:
auto_x_create_x_topics_x_enable: true
controller_x_quorum_x_voters:
default_x_replication_x_factor: 2
inter_x_broker_x_listener_x_name: BROKER
listeners: BROKER://0.0.0.0:9092
listener_x_security_x_protocol_x_map: BROKER:SSL
@@ -18,7 +19,7 @@ kafka:
node_x_id:
num_x_io_x_threads: 8
num_x_network_x_threads: 3
num_x_partitions: 1
num_x_partitions: 3
num_x_recovery_x_threads_x_per_x_data_x_dir: 1
offsets_x_topic_x_replication_x_factor: 1
process_x_roles: broker
@@ -30,7 +31,7 @@ kafka:
ssl_x_keystore_x_password: changeit
ssl_x_truststore_x_location: /etc/pki/java/sos/cacerts
ssl_x_truststore_x_password: changeit
transaction_x_state_x_log_x_min_x_isr: 1n
transaction_x_state_x_log_x_min_x_isr: 1
transaction_x_state_x_log_x_replication_x_factor: 1
client:
security_x_protocol: SSL

View File

@@ -26,6 +26,11 @@ kafka:
title: auto.create.topics.enable
forcedType: bool
helpLink: kafka.html
default_x_replication_x_factor:
description: The default replication factor for automatically created topics.
title: default.replication.factor
forcedType: int
helpLink: kafka.html
inter_x_broker_x_listener_x_name:
description: The name of the listener used for inter-broker communication.
title: inter.broker.listener.name