With two independent checks now writing to the same log, messages like
"system uptime only N seconds does not meet 1800 second requirement" were
ambiguous about which check they came from. Prefix every line with a
[minion-restart-check] or [boot-highstate-check] tag and reword the uptime,
threshold, and healthy messages to say what was evaluated and why it was
skipped.
Restructure the boot-highstate check from a nested if into an if/elif chain so
each outcome (restart already queued, uptime too low, healthy, already running,
forcing) logs its own reason instead of silently doing nothing.
Add a second, independent trigger to the every-5-minute health check: if the
host has been up >= 15 minutes (HIGHSTATE_UPTIME_REQ) and no highstate has
completed since this boot (lasthighstate mtime older than boot time), run
salt-call state.highstate. This recovers a host whose boot highstate
(so-boot-highstate.service) failed or was skipped, even while the minion is
otherwise healthy and touching state-apply-test.
The new path deliberately does not enable highstate, so a soup-disabled
highstate is respected and never forced mid-upgrade. A saltutil.running guard
plus queue=True prevents stacking across successive cron runs, and a RESTARTED
flag suppresses the new block when the existing minion-restart path already
queued a highstate.
When the minion is deemed hung and restarted, wait for it to become
ready via so-salt-minion-wait, then kick off salt-call state.highstate
(queued, backgrounded) so the box re-applies its states and recovers on
its own rather than waiting for the next scheduled highstate.
highstate_interval_hours describes the per-minion highstate schedule, not the
active-push pipeline, so relocate it from salt.auto_apply to a new salt.schedule
settings subtree. Repoint so-salt-minion-check at the new pillar path (it had
been left on the stale global:push path) so its restart grace period tracks the
schedule again.
- Add salt.schedule.highstate_interval_hours to defaults.yaml/soc_salt.yaml and a
side-effect-free salt/salt/schedule.map.jinja (SCHEDULEMERGED), matching the
*MERGED map convention. Consumers read SCHEDULEMERGED.highstate_interval_hours.
- Split salt/schedule.sls into salt/salt/highstate_schedule.sls (every minion) and
salt/salt/push_drain_schedule.sls (managers); update top.sls to apply the
highstate schedule via '*' and the drainer schedule via the configured-manager
block. Remove the now-empty schedule.sls aggregator.
- pillar_push_map.yaml and so-push-drainer: comment/doc updates only.
- schedule highstate every 2 hours (was 15 minutes); interval lives in
global:push:highstate_interval_hours so the SOC admin UI can tune it and
so-salt-minion-check derives its threshold as (interval + 1) * 3600
- add inotify beacon on the manager + master reactor + orch.push_batch that
writes per-app intent files, with a so-push-drainer schedule on the manager
that debounces, dedupes, and dispatches a single orchestration
- pillar_push_map.yaml allowlists the apps whose pillar changes trigger an
immediate targeted state.apply (targets verified against salt/top.sls);
edits under pillar/minions/ trigger a state.highstate on that one minion
- host-batch every push orchestration (batch: 25%, batch_wait: 15) so rule
changes don't thundering-herd large fleets
- new global:push:enabled kill-switch tears down the beacon, reactor config,
and drainer schedule on the next highstate for operators who want to keep
highstate-only behavior
- set restart_policy: unless-stopped on 23 container states so docker
recovers crashes without waiting for the next highstate; leave registry
(always), strelka/backend (on-failure), kratos, and hydra alone with
inline comments explaining why
Josh Patterson
Jason Ertel
Mike Reeves
DefensiveDepth
reyesj2
Mike Reeves
Doug BurksandGitHub
Wes