Files
securityonion/salt
Josh Patterson 6abf382ea8 so-grid-highstate: wait for fleet recovery and quiet the expected salt-upgrade warning
Two refinements to the --salt-upgraded path, surfaced testing a salt downgrade+soup
on a manager+heavynode grid:

- The post-upgrade settle loop settled on any stable reachable count >0, so when a
  target was briefly down for its salt-minion restart it could settle on the
  not-yet-restarted subset (observed: 'settled at 1' with 2 accepted) and release the
  tiered pass before nodes reconnected. Capture the reachable count just before the
  pass and wait for it to recover to that count (up >= pre-upgrade target) and hold
  steady, with an initial grace so the delayed restart dip is observed rather than
  skipped. Still compares against the pre-upgrade reachable set, not accepted keys, so
  an intentionally powered-off node never stalls past the backstop.

- The salt-upgrade pass returns non-zero by design (targets restart salt-minion
  mid-run), but it logged the generic 'nodes it missed will converge on the scheduled
  highstate' warning, which reads like a real failure. Mark that dispatch as
  expect_restart so it logs a benign, explanatory line instead.

Verified live: with the heavynode's salt-minion bounced during the settle window, the
loop logged 'fleet recovered to 2 minions up (>= pre-upgrade 2)' and only then ran the
tiered pass (heavynode highstate 427 succeeded, 0 failed).
2026-08-06 09:46:12 -04:00
..
2026-08-03 15:19:18 -04:00
2026-07-28 10:26:29 -06:00