reyesj2
eca2a4a9c8
Logstash consumer threads should match topic partition count
...
- Default is set to 3. If there are too many consumer threads it may lead to idle logstash worker threads and could require decreasing this value to saturate workers
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-08 16:17:09 -04:00
reyesj2
dff609d829
Add basic read-only metric collection from Kafka
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-08 16:13:09 -04:00
reyesj2
e960ae66a3
Merge remote-tracking branch 'remotes/origin/2.4/dev' into reyesj2/kafka
2024-05-02 15:12:27 -04:00
reyesj2
093cbc5ebc
Reconfigure Kafka defaults
...
- Set default number of partitions per topic -> 3. Helps ensure that out of the box we can take advantage of multi-node Kafka clusters via load balancing across atleast 3 brokers. Also multiple searchnodes will be able to pull from each topic. In this case 3 searchnodes (consumers) would be able to pull from all topics concurrently.
- Set default replication factor -> 2. This is the minimum value required for redundancy. Every partition will have 1 replica. In this case if we have 2 brokers each topic will have 3 partitions (load balanced across brokers) and each partition will have a replica on separate broker for redundancy
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-02 15:10:13 -04:00
reyesj2
f663ef8c16
Setup Kafka to use PKCS12 and remove need for converting to JKS
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-02 14:53:28 -04:00
reyesj2
de9f6425f9
Automatically switch between Kafka output policy and logstash output policy when globals.pipeline changes
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-02 12:13:46 -04:00
Doug Burks
240ffc0862
Merge pull request #12915 from Security-Onion-Solutions/dougburks-patch-1
...
FIX: Improve File dashboard #12914
2024-05-02 10:44:58 -04:00
Doug Burks
0822a46e94
FIX: Improve File dashboard #12914
2024-05-02 10:42:34 -04:00
Doug Burks
1be3e6204d
FIX: Improve File dashboard #12914
2024-05-02 10:38:56 -04:00
weslambert
956ae7a7ae
Merge pull request #12909 from Security-Onion-Solutions/fix/detection_mappings
...
Update mappings for detection fields
2024-05-01 16:15:40 -04:00
Wes
3285ae9366
Update mappings for detection fields
2024-05-01 20:11:56 +00:00
reyesj2
47ced60243
Create new Kafka output policy using salt
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 14:49:51 -04:00
Josh Patterson
72b2503b49
Merge pull request #12906 from Security-Onion-Solutions/det_easr
...
Apply autoEnabledSigmaRules based on role if defined and default if not
2024-05-01 13:05:36 -04:00
reyesj2
58ebbfba20
Add kafka state to standalone highstate
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 13:03:14 -04:00
reyesj2
e164d15ec6
Generate different Kafka certs for different SO nodetypes
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 13:02:47 -04:00
reyesj2
3efdb4e532
Reconfigure logstash Kafka input
...
- TODO: Configure what topics are pulled to searchnodes via the SOC UI
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 13:01:29 -04:00
Mike Reeves
854799fabb
Merge pull request #12902 from Security-Onion-Solutions/TOoSmOotH-patch-2
...
Update config.sls
2024-05-01 12:56:04 -04:00
m0duspwnens
47ba4c0f57
add new annotation for soc autoEnabledSigmaRules
2024-05-01 12:55:29 -04:00
Mike Reeves
10c8e4203c
Update config.sls
2024-05-01 12:54:21 -04:00
Jason Ertel
05c69925c9
Merge pull request #12904 from Security-Onion-Solutions/jertel/wf
...
mark detections settings as read-only via the UI
2024-05-01 09:54:03 -07:00
Jason Ertel
252d9a5320
make rule settings advanced
2024-05-01 12:51:04 -04:00
m0duspwnens
7122709bbf
set Sigma rules based on role if defined and default if not
2024-05-01 12:25:34 -04:00
Mike Reeves
f7223f132a
Update config.sls
2024-05-01 12:00:39 -04:00
Mike Reeves
8cd75902f2
Update config.sls
2024-05-01 11:47:51 -04:00
Jason Ertel
c71af9127b
mark detections settings as read-only via the UI
2024-05-01 11:47:38 -04:00
weslambert
e6f45161c1
Merge pull request #12900 from Security-Onion-Solutions/fix/cold_min_age
...
Cold min_age to 60d
2024-05-01 11:24:48 -04:00
weslambert
fe2edeb2fb
30d to 60d
2024-05-01 11:01:59 -04:00
weslambert
6294f751ee
Cold min_age to 60d
2024-05-01 10:59:41 -04:00
reyesj2
de0af58cf8
Write out Kafka pillar path
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 10:45:46 -04:00
reyesj2
84abfa6881
Remove check for existing value since Kafka pillar is made empty on upgrade
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 10:45:05 -04:00
reyesj2
6b60e85a33
Make kafka configuration changes prior to 2.4.70 upgrade
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 10:15:26 -04:00
reyesj2
63f3e23e2b
soup typo
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 09:54:19 -04:00
Jason Ertel
ad1cda1746
Merge pull request #12893 from Security-Onion-Solutions/jertel/wf
...
update annotations for duplication
2024-05-01 06:32:13 -07:00
Jason Ertel
66563a4da0
zeek networks will only ever have one HOME_NETWORKS setting
2024-05-01 09:31:11 -04:00
Jason Ertel
d0e140cf7b
zeek networks will only ever have one HOME_NETWORKS setting
2024-05-01 09:30:52 -04:00
Jason Ertel
87c6d0a820
zeek networks will only ever have one HOME_NETWORKS setting
2024-05-01 09:29:36 -04:00
reyesj2
eb1249618b
Update soup for Kafka
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 09:27:01 -04:00
reyesj2
cef9bb1487
Dynamically create Kafka topics based on event.module from elastic agent logs eg. zeek-topic. Depends on Kafka brokers having auto.create.topics.enable set to true
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-05-01 09:16:13 -04:00
Doug Burks
9a25d3c30f
Merge pull request #12897 from Security-Onion-Solutions/dougburks-patch-1
...
FEATURE: Lower EVAL memory requirement to 8GB RAM #12896
2024-05-01 08:01:20 -04:00
Doug Burks
9a4a85e3ae
FEATURE: Lower EVAL memory requirement to 8GB RAM #12896
2024-05-01 07:54:38 -04:00
reyesj2
bb49944b96
Setup elastic fleet rollover from logstash -> kafka output policy
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-30 16:47:40 -04:00
Jason Ertel
72db369fbb
Merge branch '2.4/dev' into jertel/wf
2024-04-30 15:16:41 -04:00
Jason Ertel
84db82852c
annotation updates for custom settings
2024-04-30 15:14:56 -04:00
reyesj2
fcc4050f86
Add id to grid-kafka fleet output policy
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-30 12:59:53 -04:00
reyesj2
9c83a52c6d
Add Kafka output to elastic-fleet setup. Includes separating topics by event.module with fallback to default-logs if no event.module is specified or doesn't match processors
...
Signed-off-by: reyesj2 <94730068+reyesj2@users.noreply.github.com >
2024-04-30 12:01:31 -04:00
coreyogburn
ea4750d8ad
Merge pull request #12882 from Security-Onion-Solutions/cogburn/community-repos
...
Mark Repos as Community
2024-04-30 09:12:25 -06:00
Doug Burks
e9944796c8
Merge pull request #12886 from Security-Onion-Solutions/dougburks-patch-1
...
FIX: Elasticsearch min_age regex #12885
2024-04-30 10:26:04 -04:00
Doug Burks
4d6124f982
FIX: Elasticsearch min_age regex #12885
2024-04-30 10:18:34 -04:00
Jorge Reyes
dd168e1cca
Merge pull request #12881 from Security-Onion-Solutions/2.4/finalpipefix
...
Update expected timestamp format in final pipeline for system events
2024-04-30 09:39:18 -04:00
Corey Ogburn
ddf662bdb4
Mark Repos as Community
...
Indicate that detection rules pulled from configured repos should be marked as Community rules.
2024-04-29 16:22:30 -06:00
