zeekctl's post-terminate archives the final logs in the background and returns
immediately unless StopWait is set, so the container exits and takes the archiving
with it, stranding unarchived logs in /nsm/zeek/spool/tmp on every restart. Docker's
default 10s grace is also too tight for the entrypoint's SIGTERM trap; overrunning it
means SIGKILL and crash directories on the next start.
Both are needed. StopWait alone gives the stop more work to do inside the same 10s
window, which was measured ending in SIGKILL with logs stranded in the spool.
disabled.sls used docker rm -f, which never delivers SIGTERM, so stop the container
before removing it.
Reported in discussion #16174.
LogExpireInterval, StatsLogExpireInterval and CrashExpireInterval are only acted
on by "zeekctl cron", which nothing in the grid ran, so setting them in SOC did
nothing. Add so-zeek-cron and run it every 5 minutes, the interval upstream
recommends.
This also restarts a node that died unexpectedly and marks it crashed so a crash
report is written, which is what CrashExpireInterval then reaps.
The crontab runs as root because the script needs the docker socket; it drops to
the zeek user inside the container so the stats logs and zeekctl-config.sh it
writes stay owned by uid 937.
Annotate the five zeekctl settings that were previously undocumented. The regex
on LogExpireInterval matters: a bare number means days, and a value shorter than
LogRotationInterval raises ConfigurationError, which fails the zeekctl deploy in
the container entrypoint. Zeek then never starts while Salt still reports success
and the container still reports healthy. Excluding the min unit keeps that
unreachable at the default 3600 second rotation interval. MinDiskSpace and
MailHostUpDown only send mail and the image has no sendmail, so their
descriptions say they currently have no effect.
Josh Patterson