Run zeekctl cron so the expire settings take effect

LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval are only acted
on by "zeekctl cron", which nothing in the grid ran, so setting them in SOC did
nothing. Add so-zeek-cron and run it every 5 minutes, the interval upstream
recommends.

This also restarts a node that died unexpectedly and marks it crashed so a crash
report is written, which is what CrashExpireInterval then reaps.

The crontab runs as root because the script needs the docker socket; it drops to
the zeek user inside the container so the stats logs and zeekctl-config.sh it
writes stay owned by uid 937.

Annotate the five zeekctl settings that were previously undocumented. The regex
on LogExpireInterval matters: a bare number means days, and a value shorter than
LogRotationInterval raises ConfigurationError, which fails the zeekctl deploy in
the container entrypoint. Zeek then never starts while Salt still reports success
and the container still reports healthy. Excluding the min unit keeps that
unreachable at the default 3600 second rotation interval. MinDiskSpace and
MailHostUpDown only send mail and the image has no sendmail, so their
descriptions say they currently have no effect.

salt/zeek/config.sls

@@ -133,20 +133,6 @@ zeekctlcfg:
     - defaults:
         ZEEKCTL: {{ ZEEKMERGED.config.zeekctl | tojson }}
 
-# ZeekControl only acts on these five settings from its 'cron' command, so track them
-# separately from zeekctl.cfg. Changing CompressLogs or LogRotationInterval rewrites
-# zeekctl.cfg but must not trigger a cron run, so zeekctlcron watches this file instead.
-zeekctlcronsettings:
-  file.managed:
-    - name: /opt/so/state/zeek/zeekctl_cron_settings
-    - makedirs: True
-    - contents: |
-        LogExpireInterval={{ ZEEKMERGED.config.zeekctl.get('LogExpireInterval', '') }}
-        StatsLogExpireInterval={{ ZEEKMERGED.config.zeekctl.get('StatsLogExpireInterval', '') }}
-        CrashExpireInterval={{ ZEEKMERGED.config.zeekctl.get('CrashExpireInterval', '') }}
-        MinDiskSpace={{ ZEEKMERGED.config.zeekctl.get('MinDiskSpace', '') }}
-        MailHostUpDown={{ ZEEKMERGED.config.zeekctl.get('MailHostUpDown', '') }}
-
 # Sync node.cfg
 nodecfg:
   file.managed:

salt/zeek/disabled.sls

@@ -23,6 +23,11 @@ zeekpacketlosscron:
     - identifier: zeekpacketlosscron
     - user: root
 
+zeekctlcron:
+  cron.absent:
+    - identifier: zeekctlcron
+    - user: root
+
 {% else %}
 
 {{sls}}_state_not_allowed:

salt/zeek/enabled.sls

@@ -71,25 +71,6 @@ so-zeek:
       - file: zeekctlcfg
       - file: zeekbpf
 
-# LogExpireInterval, StatsLogExpireInterval, CrashExpireInterval, MinDiskSpace and
-# MailHostUpDown stay inert until 'zeekctl cron' runs, so run it once whenever one of
-# them changes. --no-watch skips ZeekControl's crashed-node watchdog, which would
-# otherwise race the container restart that the same config change triggers, and which
-# would duplicate the healthcheck beacon in salt/_beacons/zeek.py. The retry covers the
-# startup deploy still holding the ZeekControl lock. Run as the zeek user, as
-# salt/_modules/zeekctl.py does, so the stats logs and zeekctl-config.sh this writes stay
-# owned by uid 937 rather than root.
-zeekctlcron:
-  cmd.run:
-    - name: docker exec so-zeek runuser -l zeek -c '/opt/zeek/bin/zeekctl cron --no-watch'
-    - onchanges:
-      - file: zeekctlcronsettings
-    - require:
-      - docker_container: so-zeek
-    - retry:
-        attempts: 5
-        interval: 30
-
 delete_so-zeek_so-status.disabled:
   file.uncomment:
     - name: /opt/so/conf/so-status/so-status.conf
@@ -106,6 +87,21 @@ zeekpacketlosscron:
     - month: '*'
     - dayweek: '*'
 
+# LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval are only acted on by
+# 'zeekctl cron', so run it on the interval upstream recommends. This also restarts any
+# node that died unexpectedly. Runs as root because the script needs the docker socket;
+# it drops to the zeek user inside the container.
+zeekctlcron:
+  cron.present:
+    - name: /usr/sbin/so-zeek-cron > /dev/null 2>&1
+    - identifier: zeekctlcron
+    - user: root
+    - minute: '*/5'
+    - hour: '*'
+    - daymonth: '*'
+    - month: '*'
+    - dayweek: '*'
+
 {% else %}
 
 {{sls}}_state_not_allowed:

salt/zeek/soc_zeek.yaml

@@ -59,23 +59,49 @@ zeek:
         description: This setting enables compression of Zeek logs. If you are seeing packet loss at the top of the hour in Zeek or PCAP you might need to disable this by seting it to 0. This will use more disk space but save IO and CPU.
         helpLink: zeek
       LogExpireInterval:
-        description: Number of days to keep rotated Zeek logs in /nsm/zeek/logs, or 0 to keep them forever. Saving this setting runs "zeekctl cron", which deletes any logs already older than the value you set. Note that this is a one time cleanup rather than a rolling retention policy, so logs that age past the interval afterwards are not removed until this setting is changed again. Ongoing cleanup based on disk usage is handled separately by so-sensor-clean.
+        description: >-
+          How long to keep rotated Zeek logs in /nsm/zeek/logs. A bare number means DAYS, so 7 means 7 days.
+          You may also give an explicit unit, such as "7 days" or "12 hr". Use 0 to keep logs forever.
+          This value must not be shorter than LogRotationInterval (3600 seconds by default), so the smallest
+          usable value is 1 hr - Zeek will fail to start if it is shorter. Expiry is applied by "zeekctl cron",
+          which runs every 5 minutes, and removes log files older than this based on their modification time.
+        regex: ^(0|[1-9][0-9]*( ?(day|hr)s?)?)$
+        regexFailureMessage: Enter 0, or a positive number optionally followed by "day" or "hr" (for example 7, "7 days", or "12 hr"). Minutes are not accepted because a log expire interval shorter than the log rotation interval prevents Zeek from starting.
         helpLink: zeek
         advanced: True
       StatsLogExpireInterval:
-        description: Number of days to keep entries in the Zeek stats log, or 0 to keep them forever. Saving this setting runs "zeekctl cron", which applies the new value once. See LogExpireInterval for details on this behavior.
+        description: >-
+          Number of days to keep entries in the Zeek stats log, or 0 to keep them forever.
+          Applied by "zeekctl cron", which runs every 5 minutes.
+        regex: ^[0-9]+$
+        regexFailureMessage: You must enter a whole number of days, or 0 to keep entries forever.
         helpLink: zeek
         advanced: True
       CrashExpireInterval:
-        description: Number of days to keep Zeek crash directories, or 0 to keep them forever. Saving this setting runs "zeekctl cron", which applies the new value once. See LogExpireInterval for details on this behavior.
+        description: >-
+          Number of days to keep Zeek crash directories, or 0 to keep them forever.
+          Applied by "zeekctl cron", which runs every 5 minutes.
+        regex: ^[0-9]+$
+        regexFailureMessage: You must enter a whole number of days, or 0 to keep crash directories forever.
         helpLink: zeek
         advanced: True
       MinDiskSpace:
-        description: Percentage of free disk space below which ZeekControl reports a warning, or 0 to disable the check. This check only runs during "zeekctl cron", which Security Onion runs when you save this setting. It does not delete anything.
+        description: >-
+          Percentage of free disk space below which ZeekControl reports a warning, or 0 to disable the check.
+          Note that this setting only sends email and the Zeek container does not include a mail program,
+          so it currently has no visible effect. It never deletes anything - disk based cleanup is handled
+          separately by so-sensor-clean.
+        regex: ^([0-9]|[1-9][0-9]|100)$
+        regexFailureMessage: You must enter a percentage between 0 and 100.
         helpLink: zeek
         advanced: True
       MailHostUpDown:
-        description: Set to 1 to send mail when a Zeek node changes between the up and down states. This check only runs during "zeekctl cron", which Security Onion runs when you save this setting, so it is not a continuous monitor. Requires MailTo to be set to a working address.
+        description: >-
+          Set to 1 to send email when a Zeek node changes between the up and down states.
+          Note that this setting only sends email and the Zeek container does not include a mail program,
+          so it currently has no visible effect.
+        regex: ^[01]$
+        regexFailureMessage: You must enter 0 or 1.
         helpLink: zeek
         advanced: True
   policy:

salt/zeek/tools/sbin/so-zeek-cron

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+# Copyright Security Onion Solutions LLC and/or licensed to Security Onion Solutions LLC under one
+# or more contributor license agreements. Licensed under the Elastic License 2.0 as shown at
+# https://securityonion.net/license; you may not use this file except in compliance with the
+# Elastic License 2.0.
+
+# Run zeekctl's periodic maintenance tasks. This is what actually enforces
+# LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval - without a periodic
+# 'zeekctl cron' those settings are inert no matter what they are set to.
+
+# This also restarts any node that died unexpectedly, and marks it crashed so a crash report
+# is produced. That is upstream's default cron behavior and it recovers a single node in
+# place. The beacon in salt/_beacons/zeek.py is the only other recovery path, it is disabled
+# by default (healthcheck:enabled), and it removes and recreates the whole container, so
+# letting zeekctl handle a single dead worker avoids the heavier restart.
+
+if ! docker ps --filter name=so-zeek --format '{{.Names}}' | grep -q '^so-zeek$'; then
+  exit 0
+fi
+
+# Run as the zeek user so the stats logs and zeekctl-config.sh this writes stay owned by
+# uid 937 rather than root.
+docker exec so-zeek runuser -l zeek -c '/opt/zeek/bin/zeekctl cron'